That depends entirely on where you were!
I've been in top shape and then got injured and took 3-4 weeks off and it took only 2 weeks to get back to where I was after that.
Part of it also depends on how old you are. Current research suggests that the older you are (between ages 40-60 if I remember correctly) the LONGER you retain gained fitness.
I would also take the doctor's advice with a grain of salt. I'm not a doctor and don't pretend to know more than they do, but with regard to MY OWN PERSONAL experience dealing with MY OWN BODY, it never takes as long to heal as the doctor suggests it might. This could be either:
1) The doctor is just being overly cautious so you don't blame him if you start too soon;
2) The doctor is a dumbass;
3) I have strength and regenerative powers beyond that of mortal men.
Anyway, good luck with your recovery.