As long as you provide them a decent growing medium, decent lighting, and adequate ventilation, they should grow okay. You may want to tack up some black/white poly or mylar around the area you will really be using, to reflect light back onto the plants rather than letting it hit your crawlspace walls and be absorbed uselessly.

Also, since you have company over frequently, you'll pretty much have to have a carbon filter to get rid of the smell. This will probably be your biggest problem, because without sealing it all into a room or a grow cabinet, the smell might tend to expand throughout your crawlspace and then upwards into the rest of the house.

You don't have any rooms at all upstairs, that you can reasonably come up with a reason to keep company out of? If you can use an out-of-the-way room then I'd build a cabinet or two and keep them out of sight in that room.