Hi.
Being that your babies are outdoor you can always expect them to take a role in the local ecosystem. You will never achieve a controlled environment in a field so dont stress it too bad.
Also, remember that anything you apply to the plant will make it to your lungs in some quantity.
Check the underside of the leaves for tiny little eggs. You dont want to invite a colony to set up camp so work on this first. These things you can eliminate with a citris based soap in many cases. With a solution of soap and water you can soak em down and let the soap dry the eggs out. This is not always 100% effective but I always prefer to start out using something mild. Why chem down your eventual smoke if you can avoid it.
Outdoors you will not achieve a pest free plant nor do you need to.
Leaf loss is natural and will not affect yield to any huge degree unless your babies are full blown being consumed on a huge scale.
Many times the natural predator of the pest will help out and solve the problem. Many times if you simply make it an unpleasant and hostile environment for the little pests they will simply move on to some other plant.
Try the soap out for a couple days . Dont apply too much or you can suffocate your babies. I would say use what you would for a sink of dishes...per plant. And after 24-36 hrs rinse it down so the leaves can breath. Repeat if needed and you should see a difference.

If you can, take a pic to help determine what it is eating them and how bad the damage is.
But, like I said, I prefer to start mild and only bring out the big guns if you cant solve it.

peace.