Some girls care a lot about that sort of thing, and you must be a guy who does, too, if you find it can diminish your pleasure if you encounter the same thing in a girl. Maybe consider showering together as part of the preamble to hitting the sack?

Sweat per se isn't a big turnoff for me. I sorta like it when my husband smells "manly," but I also know him real well. It'd probably be totally different with a stranger. You definitely want to believe a relatively unfamiliar person you're about to become intimate with is real clean!