Well, you totally are. It effects everything from handling to tire wear and drive train reliability. The Euros and Japanese honed in on this years ago.
The problem right now is that Toyota is basically evolved into producing bloated American cars from the 90s, while the Americans are slimming down. Everything Yota is making is ginourmous and inefficient. Even the SUVs, which are land yachts. The Tacoma, which was the de facto efficiency mobile in the truck segment gets worse fuel economy than the new full size F150's. The Camry and Corolla have gotten huge.
I was a long time Toyota fan FWIW, my first car was an old Land Cruiser, and my mom had a Tercel when I was young.
The Ford Focus is a great car all around in that segment, and the Fusion for the next size up. I know several people who were deciding between the Fusion, Mercedes C Class, and BMW 3 series. That's how far Ford has come in their consumer range.