It's probably Perlite or Vermiculite. It's a common, cheap soil medium growers use. Finding small pieces is sometimes common is mass produced crops. It's hard to maintain every single bud cropped, so some pieces get caught here and there.

Nothing to worry about. Just remove it and carry on smoking.