people will talk shit no matter what you do
you can have em talk shit with you having that girl by your side
or you can have them talk shit while you stand alone

if you like her and she you then fuck everyone and handle it

your brother will get over it
you might have to fight though