False. No one avoids it because of that. Most people are well aware of the great capabilities they bring to the fight, but most pilots are pilots because they love flying.
Most avoid it because it is not fun, is (usually) mind-numbingly boring, and is not flying.
Also false. The people operating UAVs have pushed our current technologies to the limit and are doing a great job; better things are on the way, but no one is limiting what we can do in the fight. The limitation is that what we currently have fielded only goes so far.
I take it you're talking about the 11X vs 18U divide; the real schism exists only between those who can carry out the mission, and those who cannot. Just as with pilots, there are good UAV operators and there are bad ones. It is actually beneficial for the 18U field to succed, in the eyes of the current pilot, because then less pilots will be pulled to UAV assignments. There are other issues, but it isn't one of caste.
The reality of the situation is this: being an 18U is the best desk job in the Air Force. You are operationally oriented, you affect the war on a daily basis, and then go home at night. If you have no ambition to actually fly in an airplane, it's great. If you think you might have the slightest notion to fly in an airplane, but are worried you might be in UAVs anyway, take a path through pilot training and at least give yourself the 80% chance of flying in an airplane, then do your best. Luck and timing will work the rest out. Either way, no one is out there holding back because they don't like their job.