Is it a real hermaphrodite or just a female with a few banana male flowers? Clusters of balls with male flowers are an indication of a hermaphrodite, where single male flowers can appear on a true female (XX).

These females can produce seeds that are ALL female AKA XX seeds. Problem is not all of the pollen is viable.

Back to your question;

If the hermaphrodite gets little white seeds then they can taste bad. I had a hermie plant I called "Black Spice" It went so dark red that it almost looked black to me. Amazingly stony too, but the flavor was not my favorite. Very strong and spicy almost like a dark red wine. I digress..

I think it will hurt mostly the yield.