It's a bit confusing, can't you join friends or can't you invite friends? In any case, there's a maximum number of people that can be in a town at the same time (24 or 25 if I remember correct).

When that max amount has been reached, you can't invite or join anymore until someone leaves.