Oh well, lots of failing iMacs these days...
Most important thing first: I wasn't aware that Apple made an iMac version of the GT120 GPU. Can you dump the ROM? I guess it'll be very useful here, since it's going to tell us how Apple expects an Nvidia GPU to be configured.
Secondly: It's actually a good thing that both the internal display and the external DP port are functional (did you test the latter with a real DisplayPort device or with an passive adapter?).
I don't know why your screen is garbled like that, maybe your screen is detected incorrectly (EDID issue?). Could also be an physical connection issue.
Thirdly: I had a quick look at the VBIOS. The definition of ports/connectors is a similar mess like we've seen in the neighboring thread (GTX 660M) with a total of TEN connectors: 1x LVDS, 1x HDMI, 3x DVI-D, 3x external DP, 1x internal DP, 1x VGA. And this on a GPU which can't drive more than 2 displays at a time...
After dumping the ROM from your original GT120 GPU it would be a good idea to identify the actually needed ports (I think I've described the procedure in the GTX 660M thread). All you need is covered in the official spec: http://ftp://download.nvidia.com/open-gpu-doc/DCB/1/DCB-4.0-Specification.html#_connector_table
Some time ago I made a script to dump and decrypt some of that information, which has come in handy form time to time. I guess I'll clean it up and publish it when I find the time.