Join Date: Mar 2006
Location: Calgary - Shaw phone/internet, OTA attic / Pigeon Lake - CCI Wireless, VoIP.ms, OTA, FTA, LTSS
From your description it sounds like you are running the iMac desktop version of the same hardware Apple used in my beater MacBook Pro from that era. If so, 3 GB with PC2-5300/6400 is doable.
The MacTracker application for OS X or iOS from Ian Page is a great resource to tell you quite quickly what is the real maximum configuration for your particular machine, as opposed to the official Apple maximum.
My MacBook Pro built the last week of June 2008 is still going strong running Yosemite with a Samsung 840 EVO and 6 GB of RAM.
Kind of rare these days but keep your eyes open for a PC2-6400 4GB SODIMM to max out the machine.