If you really are "slow in replenishing muscle glycogen", you may have a digestive ENZYME deficiency which is preventing you from completely breaking down carbohydrates. Normally, you'd likely have stomach or intestinal discomfort after eating (especially starchy foods) if this was the case, but it still might be a possibility. You could try taking some nutritional supplements such as Essential Enzymes (from Source Naturals, I believe) to see if the problem can be corrected. You might also try something as simple as chewing your foods more thoroughly to achieve better carbohydrate breakdown.
All of this is really out of my area of expertise, so I suggest seeing an M.D. if your problems are concerning enough to warrant that much attention and if they persist after a couple of months of implementing the suggestions put forth in this thread.