I had the same problem, the little fuckers found a way in thru a part of the window that no caulking. I got these little square poison traps. Its got some poison in it and they take it back to the hive and it kills them when they eat it. Pretty neat when you think about it....

Atleast your hands would be clean...