I remember a few years back someone got sick and died from Legionnaire's disease after exposure to a bag of potting soil.
http://healthgate.partners.org/browsing/LearningCenter.asp?fileName=11709.xml&title=Legion naire's%20Disease
Also much more unlikely is Anthrax.
http://www.astdhpphe.org/infect/anthrax.html

Then again it's probably just a virus going around. Wear gloves and a dust mask if you continue to handle soils and perlite just in case.