honestly i dont think perilite would be harmful, its just expanded lava rock, so i really doubt it would be affected by any flame put to your final product...

if it happened to me i would try not to mess with it too much until harvest time, then as you're trimming leaves and whatnot you can try and pick out the chunks. that or wait until it dries