That certainly is an odd proprietary connector. If you want to cut into the FM antenna, you could add a balun to the FM coax (to convert from 75 ohm to 300 ohm) and then connect the two connectors on the balun to the two FM wires...

My stereo has a 3-pin antenna connector. The 3 pins are called, in order, "FM", "AM", and "Loop". There is an external antenna that attaches to this connector. I think the type of connector is called a "mini-DIN"?

I have no idea what would attach to that, the balun wouldn't work since it doesn't use either screws or coax to connect.

Any ideas how to connect cable?

My stereo has the exact same setup as yours. except one additional one called GND, which the owners manual tells me to connect it to earth or a copper water pipe.. my fm has a single 75 ohm lead.. while a coaxial cable also may be 75 ohms, but uses the outer shielding conductor, i have no idea where to connect the shielding conductor onto the back of my stereo without inducing more noise/interference into my FM broadcasts. My older stereo receiver (RIP Sanyo) had the 2 300ohm screw terminals, and i used to simply connect the balum thingy pictured in this thread to that and it worked no problem. I also noticed you could have the crappiest cable tv picture on your cable line and fm would still come in sounding crystal clear.