I had the same thing happen to me, though I wasn't riding a bike, and it was pencil lead. I got stabbed by it and the entire tip broke off in my leg. It was (is) about half an inch of graphite. Anyway, because I couldn't get it out, it eventually healed over. It's still there to this day, it looks kind of like a freckle though. To make a long story short, yes, get it removed. This is your only chance to get it safely removed.

EDIT: Curse my slow typing!