When you inhale and hold for a long time sometimes your lungs will soak up the visible particles completely. This can happen with grass and tobacco and probably just about anything else you can smoke if you hold it long enough. There's a name for this but I can't think of it right now.

Also, you might just be swallowing it. If you're not doing this already, try opening your nasal passage when you suck down the hit - makes it more like a normal breath so you only take it down as far as your lungs.