I'm a very social person, and I have lots of friends who do not smoke.

My husband and I toke daily, so all of our friends and adult family members know that we do, but when it boils right down to it: it's nobody's business if we're baked or not.

We go to work.
We pay our taxes.
We pay our bills.
We recycle.
We pay attention to politics.
We love being parents.
We enjoy a vast array of music, art, and film.
(The list goes on and on) and with our true friends, those are the types of things that we discuss if they happen to NOT smoke also. Because they are genuine friends we have not been judged in the least.
Perhaps you should just focus on the parts of this friendship that have nothing to do with pot, as I'm sure there are many, since you've referred to this person as a brother to you. The bottom line is that if you're an adult and you lead a righteous lifestyle then getting high falls under the category of "personal routine".