Most agile practitioners understand it takes some time for a team to "gell," or settle into the routine of working smoothly together. Until that time, the team doesn't achieve its own normal throughput or velocity. This is one of the reasons for the agile guideline to form teams for the long term and keep the same people together project after project, in contrast with the traditional approach of forming a new team for each project.
What surprises me is the widely-held assumption in the agile community that it's okay to change one or a few team members occasionally. Project managers and ScrumMasters tend not to treat this as a "new team" situation. They don't adjust their projected completion dates for delivery when they change individual team members. The assumption is that swapping out one team member doesn't disrupt the team very much. I have observed quite the opposite in working with a number of agile teams in several companies. My experience suggests that any change in team composition effectively results in a new team that has to begin learning how to work together from Square One.