I was just reading the Bill Rogers / Priscilla Welch book on masters running last night, and they mentioned the one thing that contributes more directly to age-based slowdown than any other single factor: the inexorable slow drop in your max heart rate.
As your max HR drops, the HR at which even a well-trained runner can sustain a 5K or 10K, say, also drops. I've always been able to sustain around 95% of max for most of a 5K and 92-93% for most of a 10K. But when your max HR drops from 200 (my max in my late 40s) to, say 182 (which is where my max seems to be 15+ years later at age 64), your 95% and 92% HR drops as well--as, inevitably, does the pace you're sustaining at that %.
There's just no way around this. 170, which used to be my 85% HR and something that I could sustain/average for an entire marathon, is now my 93.4% HR, which is more like something I can sustain for a 4 or 5 mile race. Exactly the same HR; much harder subjective effort.
That's what kills you, which is to say, slows you. It's not the only thing that takes you down; musculoskeletal stuff, and especially facet joint arthritis (in my case), adds a whole other level of trouble. But even when your body parts hold together and you are able to train well for a sustained period of time, the gently falling max HR thing is shadowing you.
On the other hand: I'd rather be above ground than below ground. So no worries!