Size does matter in a sense...

4-5 inches is I guess smallish average

3 or 4 would be small...

Now there's been two time in my life that I have seen really really really small penises.

One was uncircumsized and was like less than a inch

And the other one was circumsized and all it looked like hard was a little head poking out of his groin.

Needless to say...I didn't sleep with these guys.

So as long as your dick just doesn't look like a little penis head poking out of your body, you should be just fine