It's a bit like (but to a much lesser extent) Jewish sypathizers during the Nazi occupation of Europe. They were criminals for boarding Jews, but was it wrong to do so? Absolutely not! They were protecting the freedoms of other human beings that were taken away by an oppressive government.

Marijuana smokers are victims, having to hide their habit from the general public... but does that mean marijuana use is wrong?