Maybe you've got a small hernia or something along those lines.

Like the_yeag said above, it could happen after you cough hard, which could be related to the reason why the doctor checks you up at the end of the physical when he/she grabs your balls and makes you cough. There could be some kind of connection there that applies to you. When's the last time you got a physical.

Just thinking outloud...