Are you using a wireless router and a cordless phone? If so, could just be interference in the wireless-G spectrum.
Even if it was broadband VoIP, a voice call with no compression is 64 kbps. It can also be compressed but that would only make any meaningful difference if you are multiplexing a whole bunch of calls in the same packet flow (they may do that in the core of the network but not from your home gateway to the curbside box). There's overhead, for sure, but it's still 10s of kb/s versus Mb/s speeds of your access line.
Bottom line, voice traffic is not going to (significantly) affect your access rates.
Even if it was broadband VoIP, a voice call with no compression is 64 kbps. It can also be compressed but that would only make any meaningful difference if you are multiplexing a whole bunch of calls in the same packet flow (they may do that in the core of the network but not from your home gateway to the curbside box). There's overhead, for sure, but it's still 10s of kb/s versus Mb/s speeds of your access line.
Bottom line, voice traffic is not going to (significantly) affect your access rates.