I've used a Nexus One, Galaxy Nexus, Nexus 4, and Nexus 5. I skipped the Nexus S and Nexus 6, and I'll have a 6P soon.
Of the Nexus phones I've used, the Nexus 5 was definitely the best one. The Nexus One and Nexus 4 both had issues with buggy hardware. The Nexus 4 in particular (along with the Nexus 7 2013) didn't work with particular WiFi 5GHz bands and had a number of design issues (like sliding off the orb wireless charger that you could get for the Nexus 4). Oh, and I think battery life on the Nexus 4 was stunningly bad.
The Nexus One would reboot frequently when attempting to use it the car as a GPS device. It's like it would overheat and reboot, I searched the Internet and found lots of people with the same problem.
The Galaxy Nexus wasn't terrible, but the screen, while being high resolution for the time wasn't very good at low brightness settings (for reading at night). It was also disappointing that Google stopped providing software updates ~18 months after the phone was released.
The Nexus 5 is actually a really solid phone, aside from having worse battery life than I'd like.
Honestly, if I could get a Nexus 5, make it thicker so that I'd get about double the battery size (~4500mAh), then that would be a perfect phone! Oh, and double the RAM to 4GB because Java sucks.
The thing I'm most looking forward to in the Nexus 6P is the rapid battery charging. The thing I'm looking forward to the least is the larger screen. I was considering a Nexus 5X, but it would take longer to ship, and the battery size improvements over the Nexus 5 just wasn't all that much. If the 5X had 3GB of RAM like the 6P I would have probably gone with that.