Yes, deer seem like a big threat. Also, it seems like they follow the same path through our woods and fields, so I don't think there's too much you can do if your plants are near their instinctive pathways.

Anyway, I think another big one will be insects.