This is something that has bothered me most of my life since I'm very short (5'3 1/2") but I just don't care anymore what women think of it. I know women prefer taller guys, but to be honest they don't even seem to care that I'm short anymore and I'm only 17 years old! It's probably a much smaller issue for older people too, and still at my age I don't have much of a problem with women since most of the ones I'm interested in are all around my height anyway.

I would say that as long as your close to their height then most of them don't care very much at all. Except the shallow ones of course.