Sounds like those parents should have never had kids in the first place. When I played sports as a child I almost always had at least one parent at games watching. There were several other kids who's parents I never saw. Those are the parents who are shocked when their kids start skipping school and getting into trouble. At least your there for those kids, even if its not fair to you.
Maybe you could send a letter home with the kids asking for more parents to help out or something