talk with him anyway, tell him you feel like your being used.
im not sure if this will help, but use nothing but "I" statements, dont include the word "you", thats a blaming word.
my therapist taught me this, its good for getting your feelings out w/o starting a fight