It would be OK for a little while to have just a fan blowing on your plants, but as the plants get bigger and release more moisture into the air, your whole room will become humid and damp, your walls and ceiling etc. will be wet and mouldy. It is far better to have an extractor fan, it will completely remove the stale, humid air from the room where your plants are. It will also help with the smell, it won't remove the stench it completely but it will help. The first time I grew cannabis I had the same sort of set up as you're talking about now (just a fan blowing on the plants) and I'll tell you what, the whole house fucking stunk to high heaven, it was really bad, we couldn't have visitors to the house or anything. This is why it's important to have an extractor fan. The fan I have now is a regular bathroom fan which takes the air through ducting and outside the house through a hole I drilled in the wall with a core drill bit. The fan is pretty quiet, I can't really hear mine if my door is shut, and if your grow-room/closet is a solid structure then the noise will be reduced even more. My grow-room is built from thin wooden batons and hardboard so it vibrates a little when the fan is on.