Select the edges along the top and use the Shift+Move technique to create new sets of faces. Make sure the center verts stay at x:0 so that the shirt won't end up with holes in it when you apply symmetry to it.

When you go into perspective view you'll probably see some problems - certain parts may not have enough edges and others may have too many. To remove a row of edges first select the edges and click the remove button, but THEN you need to go to vert select mode and remove the left over verts that used to make up the edges. Just select any isolated verts and click the remove button.

To add in new rows of edges, just select the edges you want to make a new edge between and click the connect button.

Instead of just continuing upwards, at this point I decided to get the neck setup so that I could build between the two.

I went into polygon select mode, selected the top row of polys and held down SHIFT and moved them up towards the neck.

Instead of extruding new polys, like what happens when you hold shift on edges, this simply makes an exact copy of the polys you had selected.

I adjusted the verts so they lined up in the front and side viewports and then went into perspective view and cleaned up their locations.

In addition to basic clean-up, I also removed a few edges (remember when you remove an edge, you also need to go in and remove the verts too). The amount of edges that were present lower in the torso weren't necessary for the collar.

At this point I did two things to help me continue working. I applied symmetry so I could judge the total mass better, and I applied smoothing groups so that it wouldn't be faceted anymore. Just do it the exact same way we did above (For smoothing, just select all of the polys and pick a smoothing group number; for symmetry click on the Modifier List drop-down and scroll towards the bottom where it says Symmetry).

Now to fill the gap between the bottom of the shirt and the collar. Select the bottom portion of the shirt's top row of edges and hold shift and pull them up. Adjust the verts in the front and side views so that they come to the bottom of the armpit.

Next select just the front and back edges, and skip the edges that make up the bottom of the arm hole. Hold shift and pull just those edges up. Do it again another one or two times as needed, checking both front and side views to line up with the reference, and then the perspective view to more accurately shape it as needed.

Doing the shoulders and connecting to the collar can be tricky. Instead of bring up an entire row of edges, I selected JUST the edge circled below. Same as always, just hold down SHIFT and drag the edge to create new faces. Then in vertex mode, I clicked the Target Weld button and dragged the verts along the top of those edges to the base of the collar.

For the two edges shown selected in the image, I used Bridge. When you have two edges selected and you click Bridge it will create a new poly between the two edges.

Use Bridge on the remaining edges on the front of the shirt and in the back where you can to fill the gaps. Weld any verts together that you need to, and cut in any new edges where needed to make it work

For this part, I find it's best to just work in the perspective viewport so you have a better feeling for what you need to fix and adjust.

For the Back, the number of edges didn't match perfectly so I had to do some cutting to make it all fit correctly.

Now is a good time to look your model over and see if there are any areas that you can optimize. Any place where you've got several edges close together where you could probably do just as good with fewer. Just remember that any edges you remove, you also have to remove any isolated vertices by selecting them afterwards and clicking the remove button.

If you need to, select all the polys and set them all to the same smoothing group again. Using Bridge tends to add extra smoothing group info that you can get rid of by clicking Clear All and then assigning a smoothing group to everything again