i think, as worker, you have the right to leave when you want to go get it checked out, its your body not his, who is he to say if you're okay or not. i hardly think he has a Ph.D. and therefore is not qualified to tell you if you're okay or not.
you should definately get it checked out. what kind of boss is that, that is just stupid.