Nothing should come between love and attraction. If your brother doesn't like it, then that's too bad. Wait a little bit to tell your family, if they're really all for your happiness, then they'll be glad.

Go for it.