I will go out on a limb here and say that the problem is not likely to be the foods you eat, cardiovascular fitness, or allergies, but excess weight. Imagine strapping the weight you recently lost onto your bike or in your pack and going for a ride. If you thought your fitness level was bad now, it will much much worse carrying around an extra 80 lbs. that does nothing but slow you down.
I asked the same questions as you, tried to increase my cardio, changed my diet, and nothing really helped me become fitter until I shed an extra 35 lbs. I made very small incremental changes in my diet at monthly intervals (mostly portion control), and tried to make those changes a permanent habit. Increased exercise did not help that much -- about 2 lbs lost initially. I think the key is not to make large or many changes quickly, because those changes are not likely to become habits over the long term.