You should be able to watch the growth of the pollen sacks. You can even pick them off to delay them, but if you aren't sure of the sex yet, don't chop them down. You have at least a day (more likely a few) to see the growth of pollen sacks (which is unmistakable) before you need to play lumber jack.