Sounds like a Catapiller to me. It wouldn't be a worm as all worms are earth creatures and I highly doubt they would, or could, climb into a bud.

I've seen crops with catapiller infestations before. The eat into the center of your buds and leave a nice trial of poop for you. They can ravage a whole crop within days by simply eating them. A major pain in the ass.

Look for a new connect. If I found any kind of bugs in my weed, I would find someone new and better because that is just nasty man. Imagine if you started smoking it. Looking down at the bowl and seeing alittle worm wiggling around.