So, I got this to work with a Sapphire Vapor-X HD 6750 1gb DDR5 card. Turns out the 6750 and 5750 are the same cards (as are the 6770 and 5770).
I'm running 10.8.0 (though I will upgrade to 10.8.2 after creating a backup and make sure it still works) on a Mac Pro 4,1.
Basic steps were:
0) Boot into Windows, use GPU-Z to check the details of your GPU -it should match this, more or less (mine reported a different die size):
http://www.techpowerup.com/gpudb/249/ATI_Radeon_HD_5750_1_GB.html1) Save a ROM image, then make a backup copy.
2) Use RadeonBiosEditor to change the device type to a 5770 (*see notes below)
3) Follow the "Make your own ROM" instructions in noradnija's link
4) (This worked without problem for me) After finishing that, you have to add the device ID to the ATI5000 kext and AMDAccelerator kext (I'm not by a Mac right now so I can't get the exact filenames)
5) Repair permissions and rebuild kext cache - I used Kext Wizard
6) Reboot and everything worked for me - DVD player and Steam games, QE/CI as tested by Desktop ripples... supposedly you can get a boot screen via a DVI->VGA cable, though I haven't tested that yet (will test and report in later). Card IDs as a 5770 in System Profile.
PROBLEMS:
Like I said, this went pretty great once I sorted it out. That being said, there were some hiccups.
1) My sound quit working after I did this. A PRAM reset fixed it straight out (boot with option-command-p-r)
2) The first time I booted up OSX saw both cards as being in an extended desktop, though the 6750 wouldn't output anything, and was the primary screen, so I couldn't log in. I'm 90% sure that this was because I didn't repair and rebuild kext's, but I didn't realize that until I had reflashed back to stock, so I ended up rebuilding the kexts, then flashing again. Worked fine after that.
3) Windows (7 Enterprise 32-bit edition) is not working with the card right now. I had it running with standard drivers before flashing and after it flashed it stopped working. I uninstalled and reinstalled the drivers and that had no effect. Device manager says that Windows has stopped this device because the drivers reported an error, and GPU-Z reports an incorrect memory speed.
I haven't fixed this yet, but I have two leads. First, I am going to wipe and reinstall my Boot Camp partition. I only use it for a couple games (and flashing this card) so no big deal there. If that doesn't work, there's some mention that changing the Manufacturer ID from AMD/ATI (1002) to Sapphire (don't know that code off hand) will fix a similar problem in one of the Hackintosh forums. I will try this when I do the 10.8.2 update as presumably I will need to re-update the kexts with device IDs again anyways and I will need to put the new device ID (based on the Sapphire manufacture code) anyways. I will report back in with what I come up with there, as well, so anyone who goes searching for this information later will have a more complete picture.
Big thanks to noradninja and jeanlain (and netkas and rominator) for everything they've posted about this process. There's NO WAY I would have been able to even attempt it without their help. I'll I've added is the confirmation that the 67xx series is the same as the 57xx (which was on Wikipedia).
