Somewhat true when considering moving to other AF platforms, at least in the last few years. We have had the occasional pilot go to a "special" flying assignment but that was a good career move. I can't think of too many in the past few years that have gone to other flying assignments without intending to return.
One thing to consider about the U-2: if you decide to do it and get accepted, you better be sure you want to be there. No one forced you to go... and you won't get much sympathy if you get there and decide you don't like it. It's a self-inflicted wound. We offer you a lot of chances to take a hard look at the good, the bad, and the ugly about U-2 Life. So get leave, buy an airline ticket, and go spend a couple of days at Beale to check it out before you apply.
p.s. Keep in mind, the U-2 Program is overmanned. Obviously. As evidenced by the fact that some genius at AFPC decided they didn't qualify for a pilot bonus. So don't come to the U-2 "for the money". And don't come there because we have the T-38A. Come there because you want to be a high-altitude Recce Pilot in the cockpit of a U-2. In a pressure suit. By yourself.
If that isn't your primary reason for going, then save everyone a lot of hassle and look elsewhere.