This is very true. I think people who say anyone can do anything if they just try have never been in classes that were actually challenging. Calc I is a class that is required for a wide variety of majors, and it is VERY common for students to struggle mightily in it even after failing it the first time around and taking it again. And the fact is, Calc I is child's play compared to even senior level math, physics, and engineering classes, let alone graduate level classes in those subjects.
Upper level classes can be difficult at times and obviously those that get there have passed the "hard" prereqs but at my university, Calc 2 and DiffEq are the notorious classes. Calc 3 is not so bad but harder than Calc 1.
I think that's how it generally goes. Calc 2 was rough. DiffEQ was very demanding and is in general pretty notorious, but I found it doable with lots of practice. Calc 3 was pretty medium difficulty... Linear Algebra was easy even though it was a higher numbered class than Calc 2 or DiffEQ at my school. I took AP Calc in high school and thought it was easy. Scored a 5 on the AB test. Same with AP Physics. I had excellent teachers though.