Hi. this is a method i pretty much burnt my brain out trying to figure out. the method described here is to get a raytraced depth map for post-processing dof, which up til now is not possible. this is a post-processing method for use with post-processing filters, such as potoshop cs' lens blur.this tutorial is made using 3ds max 6, but it can literally be done in ANY 3d software that produces depth maps (sometimes called z-depth, z-buffer etc...) AND reflect refract passes. i've never used any other 3d program so i wouldn't know the process for those. you just need the greyscale z-map and reflect refract passes. Here is our image. don't tackle me with envy. it's a quicky to demonstrate this process.this tutorial is made using 3ds max 6, but it can literally be done in ANY 3d software that produces depth maps (sometimes called z-depth, z-buffer etc...) AND reflect refract passes. i've never used any other 3d program so i wouldn't know the process for those. you just need the greyscale z-map and reflect refract passes.
Above is our image. don't tackle me with envy. it's a quicky to demonstrate this process.
Step one is to prepare the renders. you need to take any object, that is in the reflection/refraction's path and assign each a unique color. not just any color but a blown out color. for example, make your diffuse a 100% blue color. now, add a gradient map to the self-illumination slot (max). both colors in the gradient should be the same exact color as the diffuse. i say "both" but some gradients have more then one color. anywho, the gradient is not used for it's gradiation effect but (in max) for the output function. therefore, you can use any map that allows you to pick the colors used AND has an output function. other 3d softwares must have something similiar, you just need to find out how. ok. now, turn up the output amount to like 6 or more. this ensures that your object will be easily distinguishable from any other object in the scene. repeat this for every object that's being reflected/refracted, using a different color for each object. do this also to all lights and atmospherics, too. they need not be blown out but just a unique color. a blown-out color would make it denser and that would not be desireable for this since we need it's visibility unaltered for this technique. If you have fog, do the same thing, but when you composite them, the reflection goes on top of the refraction. also, turn off your environment maps and make your background black. they are unnessesary and would actually make this harder. now, set the outputs for a depth map (z-depth, z-buffer etc...), and reflect, refract passes. the reflect/refract maps must be done in two seperate passes. one more thing, when you render the z-depth and reflect/refract passes, you MUST turn off anti-aliasing as depth maps are not meant to be anti-aliased. in max, i couldn't turn it off completely without disabling the reflect/refract passes so in order to bypass this, you set the anti-alias setting to it's lowest which gives pretty much the same result: no anti-aliasing. ok. render those and don't forget the actual render to which you will apply the dof to. Now that we've got all four of our maps (diffuse, z-depth, reflections, refractions), let's go to photoshop. Take all your maps and stack them on top of each other. the software-produced depth map should be on bottom.
Basically, in the reflection and refraction passes, we are trying to isolate anything other than black and assign it a depth level.
Now, let's start with our reflection pass. hide all other layers. select the reflection layer and do a color range selection (select menu > color range). (Above image).pick a color, any color so long as it's not black and using the fuzziness slider (maybe other graphics programs have a similar feature), select all of the color you chose, without selectiong any other color (including black). in this case, i chose blue to start with. (Below Image)
Now that you've selected all of the blue in the image, without going into any other color, make a layer for it and fill the selection with white. Name it blue and hide it.
Back to our reflection pass. repeat the previous step for any other object/color in the scene, naming them accordingly.
now, if you have anything other than an object that is in the reflection that is not black, do a color selection for it and fill it with white. these are usually light reflections or any other atmospheric in the scene. use the following steps to assign them a depth level.