Absolutely
Anything can get into your closet, there isnt any way you can have a closet grow sompletly air tight and sealed up.

Just take precautions, sanitize everything, equipment, soil. Watch for first signs of pests, and deal with them accordingly early, so now harm comes to your plant.