Pick them off manually.
Place collars around the base of each plant- a party cup with the bottom cut off, slit down the side, sunk about an inch into the soil around the stem, and taped back up will prevent some types of caterpillars climbing up your plant. These are actually called 'cutworm collars', but cutworms are more likely to attack your plant at its base. What you have is probably something else entirely.