Putting worms in your pot will do no good because first of all, the plant grows faster than a few worms can compost, and secondly every time you water you will drive your worms out of the pots.

What you need is a worm bin. This is a container full of worms. You will add food for the worms like coffee grounds, vegetable scraps, eggshells, almost anything. The N, P and K are in the materials you feed the worms. The worms then compost the material, which when finished will look like dark black soil and be full of yummy nutrients for your plants. The worms will also produce a runoff of liquid (because you will occasionally have to add water to the bin) that is also beneficial to your plants. Once the bin is full of worm castings you separate the worms and the castings and start over. Use the castings as a soil ammendment.

Do a google on vermicomposting for more detailed info.