i think youre good as far as the sunlight. ive grown successfuly with that amount of direct light, probably less. your problems are probably due to over or underwatering, possibly soil nutrients not being right. dont dig up your plants. its a learning process growing outdoors you win some you lose some. there are charts online to determine whats going wrong with your plant, i suggest you check them out. whats your dirt look like? a lot of clay? i always dig a hole 1 foot in diameter and mix bagged soil with the natural.