it is bullshit.

if there are code problems, cite them for that or give them the chance to bring it up to code.

If there are other safety issues, if they have marital problems / domestic violence issues, those are all separate issues.

If these folks are just plain old unfit, then cite cause and remove the kids.

A friend of our family, now over 18, was raised his entire life in a house with an indoor grow in the basement, not a big one (~10 flowering?)

Anyway, it was a safe setup and he's a well adjusted adult now.