My friend has had this happen on multiple plants, spanning multiple grows. He has never had a pest infestation and the holes have always been from the way the plant itself grows. If you have checked for pests and have found nothing, I would not worry too much. Keep an eye on them though over time, it may be that you do indeed have a pest problem that is just not source-able.