If you had the time and money you could slam a slander case against them. Technically he's allowed to say anything that's public record. But saying you're "associated with narcotics" sounds a lot more like he's trying to defame your character, which is slander. To prove it, you just need to show that he's going above and beyond, with an attempt for it to negatively impact your employment.