Difficulty of access/irrigation.
Arid mountainous regions are going to have water problems for sure. Most of the time, it will be dry and you will have to carry in water. If there is a torrential downpour, those soils can be quite loose and if you happen to plant in a dry bed, you could lose your plants when the water rises and brings sediment and rocks down the mountain.
Plus if the trees are spaced far apart, you will be very visible carrying supplies up there.
If the woods are closed in and denser, and you live in an area with acceptable/steady precipitation, your life will be a lot easier.
Also remember that the 'good' spots are often already taken by locals, so plant many more plants than you want to harvest, in many locations, because the likelihood of planting in someones spot that they have used for years is certainly there.
Finally, if your plants are the only tender herbaceous thing in a forest of pines and desert plants, expect heavy losses to wildlife.