Baking Maps (Xnormal)
I tried multiple times to import my high poly models into Max with the intent to bake out normal and ambient occlusion maps, but the files were just too heavy for my computer to handle. With all of the pieces involved it would have been a huge headache to bring all of these pieces into Max, crunched down or not, and take the time to render out the needed textures.
The answer was to turn to a trusty freeware application, Xnormal. With the latest update, a feature has been added to take a lot of the guesswork out of the baking within the program. Tools> Ray Distance Calculator allows you find the Minimum and Maximum numbers you will use when baking your maps. This is very similar to setting up a cage in Max/Maya but can be less precise. That being said, it is also a lot faster and can probably bake out multiple tests in the time it would take Max/Maya to do just one.
Xnormal doesn't handle large, complicated meshes very well, so I split the model up into manageable pieces to get the maps baked out properly. In the end, I think my total number of pieces/bakes was around 20. Basically, I would load in a high poly model under the high poly mesh tab, load in the low poly version of the model in question under the low poly mesh tab (for example: head_hp > head_lp), go to the Calculator under the tools tab and press "Go.” Giving the program about 30 seconds to calculate, I took the Minimum/Maximum numbers provided and entered them into the ray casting limits for the low poly mesh. I also made sure that the low poly mesh had "Average Normals” selected rather than "Use Exported Normals.” The reason for this is that hard edges can cause lighting errors in your normal maps and, since essentially a normal map is a way to provide lighting information on top of a lower poly mesh, it is best to avoid hard edges that could kill that illusion. Average Normals basically applies, in Max terms, one smoothing group to your model or, in Maya terms, softens all of the edges. This is also ideally how you should be viewing your model in your 3D application and in game engines.
From there, I gave names to the textures I'd baked, such as "head_bake_” (this was very important as I was dealing with a lot of different bakes so a non-descriptive name could have ended up causing confusion) and with Normal Map and Ambient Occlusion checked, I set my output resolution to 2048x2048 (after a few smaller test bakes to make sure my raycast numbers were accurate) and clicked Generate Maps.
For my normal map settings, I used the default Tangent Space + XYZ as, from what I can tell, these are the usual settings for most apps and engines… or if they aren't, as in the cases of Max or Unreal, then the app or engines usually have options to invert the channels fairly quickly. Depending on your needs you can also alter the settings for your AO bake.
Once I had everything baked out, I cut and pasted areas that I would use from my various baked textures to create one normal map texture and one ambient occlusion texture. The end result, when applied to the character in Max, looked something like this (Fig.22).
Using my UV map and AO bake as a guide, I began blocking in colours for my character. Not only did this help me realize what the character would look like with the colour schemes I had planned, it also helped me to set up easy selections for different materials that I could use later on to help speed up the texturing process (Fig.22).
From here, with the help of much welcomed feedback from friends, coworkers and forum mates, I started playing further with different base colours (Fig.23 - Fig.25).
Once I'd settled on one, I began refining the details of my texture by subtly painting in highlights, shadows and grime into the texture. For the dirt and grime I used a series of Photoshop brushes I have collected over time or, in the case of the brass metals, used photo overlays of metal images.
While I was texturing my diffuse (colour) texture, I was also creating my spec map. Many people will create a spec map from a desaturated version of their diffuse map; while this can work, it isn't ideal. Within your diffuse map you will have colours that aren't necessarily relevant to their material property, so if you desaturate your texture and adjust it from there, the detail will be too generalized throughout your map. For example, if you have dark green metal within your texture, and you want it to be nice and glossy, and you were to take it's section of the diffuse text and desaturate it for spec map use… it would just end up being black and flat in-game. Ideally, you should create highlights for it that will only be used within the spec map, like a nice, strong, white highlight that isn't necessarily in your diffuse texture. I tend to brighten up areas in my diffuse texture that would normal receive more light and cause sharp highlights, like the skulls on Bishop's metal plates, as I find it pops the detail off of the similar coloured background. Be careful with this though as if it is too white then it will not light properly and can end up appearing blown out or flat (Fig.26).
At the later stages of my texturing, I also began to consider different rendering solutions and how I would want to present my final image. I knew from the beginning that I wanted to be more of a purist, not relying on high end rendering software but instead, presenting my final images exactly how they would appear in game. Afterall, this was a game art competition, so I chose to use the Marmoset Engine (via the Marmoset Toolbag freely distributed by 8 Monkey Labs.) So, towards the end I began quickly viewing my changes in this rather than in Max as it was closer to what my final image was going to be (Fig.27 & Fig.28).