Cornell researchers: Vaporflys improve men's times by 2.1 to 4.1 minutes in marathon, women improve between 1.2 and 4 mins

Track and Field News is a reputable source of information. They probably read the study before posting it on their web site. If it is important to you, you should take the time to read it. It is only 8 pages.

Hi Rojo, I'm one of the authors of the study and can answer any questions you or anyone else has.

We put the code and data online here:

https://github.com/joeguinness/vaporfly

One of the nice things about the researchers.one publishing platform is that experts can read the paper for free and leave their feedback, comments, and suggestions. We'd love to hear from runners and coaches.

My big question about this is whether or not the calculated results are meaningful. It wasn't clear to me if that's the case, but I don't know enough about stats to confidently answer that. I've only done 200-level stats, but I wasn't confident of what I've listed as part #3 below. I'm certainly not a statistician (M. Math, though), so maybe someone else would read the paper and say, "Yeah that totally makes sense." If I chunk the paper into 4 parts, #3 is what I'm having trouble with.

1) get lots of results data

2) run software analysis on data

3) justify why calculations may be valid

4) discussion of results

Take the men, for example. There were 308 different men identified, and 840 race results for those men. How many of those 308 men had results in both Vaporflys and non-Vaporflys? Maybe I missed that but it wasn't clear to me, and intuitively it feels like it should be most or all of them for the results to be meaningful.

If only it was that easy! Do they help? Yes, but that is overtly simplistic. Quantifying how much they help has too mainly variables in situ. It’s definitely easier for guys to say ‘I’d be 2-4 minutes faster if I could run a marathon today’ than to deal with the other factors influencing faster marathon times.

First of all, Rojo did you know that you were cited in the discussion section? Congratulations, you can put that on your CV when you go for academic promotion! Maybe you just started this thread to toot your own horn...

Second, this is an observational study. It was not a controlled experiment. While a randomized controlled trial of Vaporfly shoes would be the gold standard, it would be very expensive and difficult to do. The cheaper and easier option is to observe real world results and try to study them in a way that minimizes bias and confounding. In this case, it seems like a reasonable alternative.

They made a model that took into account "vaporfly", "runner", "marathon" and "other/residual" effects to predict marathon time. So if those are all the potential confounders, the model is perfect and you can trust their vaporfly effect. However, if you think there are other things that might go into race performance (weather, training changes, etc) or if there is too much overlap (?multicollinearity, but I don't have full understanding of that) in the variables, there could be bias.

I wonder why they used 90% confidence intervals. Usually 95% is the standard.

Finally, what does 2-4 minute improvement mean in this study? They actually are assuming that all runners get the same effect of Vaporfly (this is a flawed assumption clearly). The 2-4 minute numbers is just the range in which they are 90% confident the effect Vaporfly is within. It is NOT the range for different runners. So if you say - some runners in the study were 2.1 minutes faster, and some were 4.1 minutes faster, that is an incorrect conclusion to make from this study.

Overall, I think it is good. It is a much more reliable analysis than something done in a Letsrun article. The results also are consistent with anecdotal data and what we all have observed and seem legit.

Joe Guinness feel free to correct me if I am not making sense.

So who exactly funded the study?

Retarted study...... so guys like Kimetto or Kipsang you want to tell me that if they had run with Vaporfly they would have run 2:00 2:01 or maybe already sub 2:00.

Total rubbish!!! It s not about shoes but it s all about EPO..... wake up people!!!

I’m dubious that EPO is a massive thing in Kenya. If your hematocrit is above 50 it’s pretty much guaranteed and being people from altitude their hematocrit is likely already pretty close to that so it’s unlikely they would risk getting their hematocrit above that.

However they are most definitely roiding. You ever see how shredded those guys are? Ain’t no way Kipchoge is running 120-140mpw and keeping those shredded shoulders without some juice.

randomcoach wrote:

Take the men, for example. There were 308 different men identified, and 840 race results for those men. How many of those 308 men had results in both Vaporflys and non-Vaporflys? Maybe I missed that but it wasn't clear to me, and intuitively it feels like it should be most or all of them for the results to be meaningful.

In our dataset, there were 71 men and 56 women who switched shoes from a non-Vaporfly to a Vaporfly shoe. However, the other people in the study are helpful for sharpening the estimates of the vaporfly effect.

Here's an example, suppose that runner A and runner B both run Boston and Chicago in the same year. Runner A wears non-Vaporfly shoes in both races, but runner B switches from a non-Vaporfly shoe in Boston to a Vaporfly shoe in Chicago. This gives us a piece of information about the effect of the shoes because we can compare how much runner B (the shoe switcher) improved in Chicago relative to how much runner A improved.

The gist of our analysis is combining a bunch of these mini comparisons in an optimal way, using a technique called generalized least squares.

@peekay, this is a pretty good summary of what we did. If I were to add to that, I think that what's different about our study, relative to other studies out there, is that we took a systematic sample of people who were fast marathon runners before the vaporflys appeared on the market, and then followed their performances as the vaporflys began to be adopted. So we are focusing on elites and trying not to bias our sample with too many super-responders.

Yes, we are assuming everyone within a sex has the same vaporfly effect, which I agree is probably not true in reality. I think we don't have enough runners in our sample to reliably estimate a distribution of vaporfly effects, but it would be interesting to try it.

We discussed some possible confounders, but ultimately this is an observational study, so readers have to use their best judgment to determine whether they think that the possible confounders are important or not. We feel that the results are strong despite a few possible confounders.

We did report 90% intervals, but the results were significant at the 95% level as well.

rojo's article was pretty helpful to us because we were initially fooled by the neon pink and yellow vaporflys that a few of the elites wore.

I don’t like this study very much, and here’s why:

1) It uses a 4 year window to judge results and therefore isn’t taking into account confounding factors such as improved calorie consumption(I.e. maurten).

2) It also doesn't take into account the very strong likelihood that there is at least a little bit of a placebo effect occurring. If everyone believes they are magically going to be faster there is no way they don’t run faster. Which this could account for the huge difference in time improvement. The human brain is absolutely accounting for at least some of this improvement and to give it all to the shoes is absurd.

"The use of the Nike Vaporfly/Alphafly running shoes does not generate a historically unusual level of performance improvement when evaluating the world record improvements and when using a performance improvement index."

I didn’t read it - but when I do, I’d be curious how you handled the fact that more serious runners would be predisposed to spend more money on the vaporflys so it would already be a biased set

It's plainly obvious that vaporflys provide HUGE performance benefits.

You had me at Cornell.

govlie1 wrote:

I didn’t read it - but when I do, I’d be curious how you handled the fact that more serious runners would be predisposed to spend more money on the vaporflys so it would already be a biased set

___________________

All runners included in the dataset are sub-elite and elite, based on marathon finishing times before the Vaporfly was released (2:24:00 for men; 2:45 for women). This helps combat the super-responder problem, as Joe mentioned above.

It's possible that those who switched to the Vaporfly are more serious than those who did not, but it's unlikely because all of the runners were selected on the grounds that they're fast. People who run sub-elite times or better are unlikely to be hobby joggers.

Another poster mentioned that this study is pretty much worthless because it is observational, not experimental, etc. That's a flawed way of thinking because one of the main questions looming over the shoe technology is how the energy savings documented in controlled laboratory settings, in which small samples of runners are running a much shorter distance on a treadmill, translate into performance differences in the uncontrolled setting of an actual marathon. The key is to look at all of the data together--laboratory data, observational data, etc. This is a piece of the puzzle.

Armstronglivs wrote:

https://sportsmedicine-open.springeropen.com/articles/10.1186/s40798-020-00250-1

"The use of the Nike Vaporfly/Alphafly running shoes does not generate a historically unusual level of performance improvement when evaluating the world record improvements and when using a performance improvement index."

I would agree with this. And overall I would agree with the numbers given which as another poster pointed out, includes some room for placebo effect/mental proposition which, even though impossible to quantify, it absolutely a huge factor.

I've said this before here - I don't even believe the increased benefit of any of these products is the largest we have seen in the historical lifecycle of athletic footwear. I believe the jumps from PU and rubber midsoles to EVA (1975) was a bigger performance increase/innovation leap. The difference is that 1) we as humans typically tend to have short and selective memories and 2) the majority of us (I'm guessing the the majority of posters/readers of this website fall under the age of 55) have only lived in an era of footwear where the innovation has very much been incremental and not radical - and radical in the sense of drastically jumping performance. I can quantify this easily to prove that statement. For years and years the battleground in performance footwear was with respect to weight and who could make the lightest product, with the perception being that weight was a huge determining factor in performance. Of course it is, but we now know that 100g of weight is equivalent to only a 1% saving in running economy. So brands battled for years over 20,30, 40g of weight that was worth peanuts in terms of what we have seen in the last 2-3 years.

What I believe happened was that footwear brands got 1) stuck in this wormhole of weight saving and 2) got lazy and complacent, because all we are really seeing now is the piecing together of puzzle pieces that have existed for many many years (stiff plates, soft and resilient foams and even the science that pieces it together has been known for decades). So if anything, footwear tech and innovation just inexplicably stalled from about 1995 through to 2015 and all we have seen is just a massive catch-up and compensation for this in the last 5 years which the majority of pundits don't seem to be able to contextualize or deal with (hence the outrage over how these shoes have ruined the sport).

When you combine that with a performance milestone like the 2h marathon that the products have helped to push/break, this is where you get the outrage. But the exact same thing would have happened if Bannister had rocked up to Iffley Road all the way back in 1954 with a pair of pebax plated spikes and broken 4 for the first time. People would have been up in arms over it and negating the legitimacy of it - even though now a pebax plated spike is a common implement that nobody gives a f%ck about.

It's not the innovation and where it's put us at - it's the perception related to the jump and a jump that really should not be unexpected.

Very good post

I think people are getting confused by focusing on time instead of % improvement. It's easier to understand expressed in minutes, but clearly that isn't how improvements work (4 minutes faster at 2:45 is very different from 4 minutes at 2:05).

For a 2:30 guy, a 2-4 minute window would be a 1.3-2.7% improvement. The same % improvement for a 2:05 guy put him between 2:01:40 and 2:03:23. Now that's lining up better with what we're seeing.

Of course, a 2:05 guy might respond differently from a 2:30 guy -- his efficiency is better to start with, he's spending less time on his feet, I don't know what else.