I think you need to look at the idea of "harm" more broadly.

Some of the crimes you propose to be victimless, including prostitution and drug use, might not only be harmful to those engaged in it (prostitutes, for example, are in many cases sex slaves), but cause widespread public health problems (think of the burden to the state if it has to care for people addicted to and sickened by hard drug use).

I am a libertarian and in full support of the "don't tread on me" mentality. But for the sake of a healthy society, there needs to be a certain level of governmental oversight. I suppose defining that level is the real question. I'm just suggesting that people don't lose sight of the forest through all the trees.