I have heard that some people who are into fitness eliminate bananas from their diets. What would be the reason for this?

First of all, I am not sure what the "into fitness" phrase indicated.  Bananas seem to be one of those foods that people either love or love to hate.  Let's discuss both the benefits and perhaps what some would see as deterrents from eating bananas.

It depends upon who you ask, but nutritionists generally recommend keeping bananas in the diet, even for those trying to lose weight.  They are high-fiber, digest quickly, and provide quick and healthy energy; they are also low in sodium and high in potassium, which helps rid the body of excess water weight.  Bananas, as well as most fruits and vegetables, contain many necessary vitamins, minerals, and phytonutrients (nutrients that are beneficial for health that are only found in plant sources).  Sometimes persons who are trying to lose weight cut out bananas.  Usually those who avoid bananas are not counting calories, but rather carbohydrates.  However, a banana only contains about 80 to 100 calories, depending upon the size.  This amounts to 20 to 25 grams of carbohydrate, which would be contraindicated on a low-carbohydrate diet.  However, the most efficient way to lose body fat is to cut calories; i.e., a caloric deficit (expending more calories than that taken in) will result in a loss of body fat.  A low-carbohydrate diet will result in loss of weight because the resulting loss of muscle glycogen (a fuel source in the muscle) will cause the body to lose water.  However, the weight loss will be mostly water and not the desired body fat loss.

Sometimes bananas are vilified because they have a moderately high glycemic index (55 of a possible 100).  The glycemic index (GI) provides a measure of how quickly blood sugar levels rise after eating a particular type of food.  However, the GI does not take into account the amount of carbohydrate actually consumed.  The glycemic load (GL), which is based on both the GI and the carbohydrate content, does a better job of estimating how much a food will raise a person's blood glucose level after eating it.  The GL of the banana is only 10, which is considered to be moderately low.  Therefore, the banana does not have the effect of causing sudden large increases in blood glucose levels.

So why would some persons avoid bananas?  Quite often, bananas are associated with high-sugar breakfasts.  When combined with pastries, cereal, skim milk, and/or orange juice, the resultant glycemic load is high.  These high glycemic foods cause a rapid rise in blood sugar levels, causing insulin levels to quickly go up, with a resultant "reactive hypoglycemia," or drop in blood glucose levels.  This rapid drop in blood glucose levels results in low energy and sluggishness about an hour or so after such a breakfast.

However, the banana, on its own or combined with other foods that are not high in simple sugars (but that are composed of a balance of carbohydrates, proteins and fats), is perfectly fine in moderation.