If the small plants are the ones you are refering to, it's too young to sex 'em. (they're from seeds, not clones...right?)

I'd leave 'em alone for now. Sometimes calyx's are late in pushing-out the pistils, and can easily be mistaken as young nanners. Even on the same plant. I'd wait till you start to see little bunches of nanners,. (will usually look like a small 'bunch' of grapes) before doing anything drastic. Keep an eye on the 'nanners' though. But, worse case scenario is that you're right and they're hermie. Pluck-off the offending nanners asap, if possible. But don't panic till the plants are mature enough to really tell.