There is not a definite point when it becomes damaging/stunting. Once the plant cannot get enough water it will start to stunt. Remember what you see above the top of the pot has a distinct partner in the root system......all scrunched together.

Generally I judge time to change pots as: When the pot consumes 1/2 of it's size in water every day, (being dry the next day), it is time to enlarge pot.