I agree that this sounds like a conditioned response.

You're starting to associate the panic with the weed, perhaps.

In order to fix this, if this is indeed the problem, you should listen to the advice given above to control your set and setting firmly.

Start alone somewhere comfortable when you have nowhere else to be and no one else to answer to. Ease yourself out of it.

Don't smoke anywhere that will make you uncomfortable - I mean, I love being high, but there's a time and a place, you know?

Hope that helps - I agree, listen to some music that relaxes you, or watch the sunrise.