Dont give a fuck what people think. If you like what you're doing, and you're good at it and feel fullfilled, you dont even have to give opportunity to opinions of you and your work, good or bad. You are in charge of you and your opinion of you. And you are in charge of who you hang with, and if someone is being a dick and making inappropriate jokes, ditch 'em. Plenty of friends out there to be made, you do not have to tolerate disrespectful people.
With a business such as yours Im sure you were more than well aware of the controversey and criticism and that you would be subjected to, and thats probably not going to change for many years, so when you're feeling blue remind yourself that you knew what you were getting into and you knew you had the thick skin to be able to handle any criticism that came your way, and take comfort that you are doing what you want without apologies for it.
Good luck, I hope you feel better about everything. *hugs*