You're absolutely right when it comes to the blame game from the VOIP service providers, as I had to figure this out for myself and received no help from the provider. What I have experienced is this:
QoS on - phone works very well.
QoS off - phone does not work very well.
Good point about slowing down other network traffic during a phone call, though the low bandwidth of the VOIP doesn't seem to affect this too badly.
It's still a great way to save money, but for those not technically-minded or not wanting to deal with some hassle, it's probably best to go with the old reliable POTS phone line.