Second set was either physically removed from the pot (unlikely, you'd have to have a FURRY pest problem...), or they rotted.
I am going to go with the latter. IT sounds to me like you had hard-husked seeds. It's not uncommon for a seed's shell to be so tough that the seedling has difficulty cracking or shedding it. Since you said that the surviving seedlings still haven't shed their shells, I'd say those were some tough husks!
Next time you germinate your seeds, before you put them in to soak, scape or scuff along the sharpest edge of the seed gently with a nail file. Not too much. It will be much easier to shed. Were these commercial bagseeds by the way? Out of a bag of some form of compressed weed? A couple of those burly workhorse production strains have amazingly large and tough seeds.