Well if it was a 5 foot plant, maybe you should have said so. And if you had a right to be tromping around other peoples' yards, you might have mentioned that too. I'm not sure what the homeowner being rich or old has anything to do with it either, lol. You think elderly ladies don't grow pot?
The foliage IS very similar to astilbe. There is also another wild weed that grows in New England that can get quite tall with leaves like that.
But either way, it's NOT pot. It looks nothing like pot... Totally wrong leaf structure. You've actually seen a pot plant before, no?