if its brown its dead and no longer useful to the plant plus that far into flower is normal for those leaves to turn color and die off i personally snip them off as close to the branch as possible with no ill effects