Well I found out some bad news. I was looking at some of the afflicted areas and lo and behold I see these little wormy looking sob's! Did some research and they're called thrips. I call them pieces of shit. Did some more research and found that a product called Concern kills them and is safe for all fruit bearing edible plants. Also found where another couple people had used this same stuff and it solved the insect problem right away. So luckily they had some at the garden center and I just sprayed them down. So we will see. Here's a few pics of what they look like. From what the experts say, these little suckers are a bad deal if you don't catch them right away. Looks like I, beign the worry wart that I am, did! So if you see little yellow spots on your plants, don't be so quick to sluff it off on some fert problem or anything else far simpler. Really look at it and see if it's these suckers. This pic is on one in the larva stage, which is where the ones on my plant were at, but they transform into a winged biting little monster that will ruin your entire grow if let go too long.

These things are really small. Less than 1/16" long in the larva stage.