It's ok to remove leaves if they are not green anymore. I would remove the all yellow leaves from my plant and they always easily come off with a gentle tug.