Before I was a stoner and smelled weed or knew someone was smoking, I never looked at them or made any mention that I noticed them or the smell.

People that don't do drugs DO notice, they just don't say anything because there is something about the unknown (in this case, the unknown being you and your drug usage) that makes them uneasy and paranoid.

I felt really uneasy around drug users and stoners before I did pot.

So, they probably do notice, just don't say anything.

And as for the watching you quote..

I look at it this way, average people don't look at "suspicious" people and automatically think "hey, he must be stoned!"

It's probably one of the last things that comes to your mind unless you've been stoned before and do smoke marijuana.