I think it really depends on the lighter. I had a crappy disposable thing that I picked off the pavement, and every time I lit it the smell was intense.

Unburned butane does have a taste, it is hard to explain it, but I have directly inhaled it straight from the refill can and it had no effect at all.