It should create a channel called Alpha 1. The background should be black, and the area you had selected should be white. Click on the Alpha channel to activate it. You can paint directly on the channel with the paintbrush tool just like a normal layer, except it will actually be affecting the opacity alpha instead.
Use the paintbrush tool with opacity around 50% and with small brush size and modify the hair's alpha to be more 'hair like'.
To go back to viewing the regular image, just click on the RGB channel. If you make additional selection with the pen tool and then convert to a selection you can add it to your existing alpha by just selecting the alpha channel, setting your color to white, and choosing Image > Fill.
Once your texture is done, Flatten it (Layer > Flatten Image) and save it as a .tga file. IT IS IMPORTANT that you save it as a .tga file. Tga files save alpha information, so it will keep the black and white alpha channel you created.
Now go to 3dsmax and open the material editor (m) and pick an unused slot (one of the grey spheres). Click on the box next to Diffuse and choose Bitmap, just like before. Find your TGA texture and click ok. Now go back to the material main options (click the button circled in the image below).
Check the 2-Sided option, and then scroll down to the rollout for +Maps and expand it.
In the Maps rollout you should see the Diffuse Color has a Check next to it, and lists a map. All of the rest should say none under Map. Click on the Map button for Diffuse Color and Drag it to the button next to Opacity that says none.
A window will pop up, make sure you set it to Copy, not Instance.
Now click on the new map in Opacity and it will let you go in and edit it.
Set the Mono Channel Out put to Alpha, and set the Alpha Source to Image Alpha. If these options are grayed out that means that the alpha channel didn't save with the TGA file. Go back to Photoshop and make sure it saves correctly.
No go back to the main material settings and click the Show Map in View port button.
Now I changed my view to the Front view and went to the create tab and made 3 plans with 0 segments.
I assigned the hair texture I just made to all 3 planes. Switch to the Perspective view port. If you just see wire frames hit the F3 button so you see the textures. If you don't seeopacity Right-click on the word Perspective in the top-left corner of the view port and go to Transparency > Best.
All three planes should now show the entire hair texture on it with opacity. What we want to do is separate it so that each plane only has one section of hair, instead of all three.
Select one of the planes, right-click on it and choose Convert to > Editable Poly from the drop-down. Now go to the Modify tab and apply an Unwrap UVW modifier to it. Click the edit button and move the UVs so that it's only over one of the hair pieces. If there is a little overlap and some of the neighboring pieces is visible, that's ok, we'll cut it out in a minute.
Once it's all set, collapse the stack.
Repeat this on all three segments. For any areas where there is overlap and you can see portions of one of the other pieces, go into vertex selection mode of the editable poly, and use Cut to cut the plane so that it's closer to the shape of the hair piece. Select and Delete any polys on the edge you don't need anymore.
Now, use Cut to cut horizontal lines through the hair. You'll need several close together at the top near the hair line since this portion of the hair will need to be able to curve more tightly then the rest.
Next do a one or two vertical cuts as well.
Now select all of your hair pieces, right-click on them and choose Object Properties from the drop-down. In the window that comes up find the Display Properties (left-hand side of window) and make sure that Backface Cull is unchecked.
Now is the hard part that will involve a lot of tweaking and adjusting on your end. There is no real step-by-step for this that I can give you, you'll just have to go with it and keep messing with it until it actually looks right.
Select one of the pieces of hair and Hold down Shift and move it over by the head. This will duplicate the object, instead of actually moving the original.
It's important that you leave your original un-altered base objects to the side so you can pick from them whenever you need to.
On the new duplicated hair piece, move, rotate, and scale it into place near the head, and then start moving verts to get it into a shape and location you like.
Now repeat this over and over with each of the pieces until it looks good. :P
Cut in / connect to create new edges whenever you need to.
Build up layers of hair to get more depth.
Work your way around the whole head.
You'll probably be constantly going back to fix and adjust the placement of various hair pieces until it looks right.
And just keep working on it until it looks the way you want it to. It can take patience and lots of tweaking, or it may just work from the get-go. It will vary.
It really helps to have a clear hair-style goal in mind, and a picture reference is always a big plus.
That's it! I rigged him, but I'm not going to do a tutorial on rigging... At least not now. Hope that this tutorial helps someone. If there was some point in the tutorial where I said "now click on ..... and blah blah blah" but never explained where ..... was or how to find it and you got totally stuck, just send me a line (and mention where it was you got stuck) and I'll fill you in, and fix the tutorial :P
I know there are typos and spelling errors and I don't care, so don't email me about them. I wrote this tutorial in Windows NOTEPAD... no spell check. And it's really long and it's too much of a pain in the ass to try and fix it now, so live with it.
Umm... that's all I can think of. Questions, comments, praise, and curses can be sent to atheynm-at-gmail-dot-com.
Tutorial Written by Athey Nansel-Moravetz, © 2007