agree. start with a plain unfertilized potting soil ( like hyponex). You can add you worm castings and perlite to it. I mix 40# potting soil, 20# composted manure, 10# worm castings, bag perlite, bag vermiculite, 1 cu ft peat moss.
Should be good neutral ph to slightly acid. But if too low of ph add lime.

got to know your garden temp.