Do what you want man but setting up camp in the middle of a crop is way risky but it would be nice to wake up to the smell of flowering buds in the morning. the leaves and acorns from the oak tree will most likely make the soil around it acidic so bring some lime with you when you first turn over the soil in your grow area. You have to lime the whole area not just the hole you dug. For stealth you should do this right before a rain because the white lime kinda stands out. Good luck, hope you crop is plentifull