Wow, that's strange. I've never had this problem before. Maybe try adding more soil mix and less perlite, like a 30% perlite 70% soil mix, and the soil mix can have humus, peat moss, etc... in the mix.

I use 1/3 vermiculite, and a 2/3 potting soil mix, that has added perlite, humus, peat moss, and one other thing that I can't remeber from the top of my head.

This may sound stupid, but you are mixing all of your soil/mediums together?

When you plant, how loose do you pack the soil? I'm thinking you are maybe not packing the soil down good enough, and that leaves spaces for the perlite to rise from when you water. Pack the mixture down good, but not so tight where water can get through, but not so light where the roots can't grab a hold of the soil and grow.

I'm guessing it isn't that otherwise you would have poor root development, resulting in shitty plants.

I don't know bro, that's all I can think of.