Hi ES

Whether or not you leave the weed residue in the mix won't affect the taste much, the 'cannabis' taste will be mostly in the butter/oil now (You'll get a taste for it I'm sure, lol).

However, remember the THC impregnated oil will still be a part of the residue, so to leave it out will potentially weaken the quality of your brownies.
In addition, the plant material will digest at a slightly different rate to the oil in the brownie, and this can help 'extend' the length of the trip.

Anyhow, here's hoping the results (whatever method you end up choosing) work out well for you. It can be a bit experimental at first, but when you crack it (and I'm sure you will) . . Wow!!