It's not uncommon for clients to ask about agile team composition. What proportion of developers, DBAs, business analysts, QA testers, and so forth works best? How does the role of the specialized QA tester change when agile methods are used? What about the role of the specialized Business Analyst? What do we do if there are fewer BAs or QA testers than developers on the team? How many experts in a given technical specialty - DBAs, network specialists, security experts, etc. - does a typical software project need? Questions of this form underscore the extent to which agile concepts have failed to penetrate management thinking.
"How do the traditional specialized roles change in an agile environment?" is the wrong question to ask. A better question might be, "What specialized roles, if any, are necessary in an agile environment?"
Role specialization encourages team members to focus on narrowly-defined pieces of the solution in isolation from one another. Working in isolation inhibits collaboration, fragments the team's understanding of the domain model, leads to hand-offs from one specialist to the next, and drives the team to depend on documents over face-to-face communication to share information. The dependence on documents leads to a higher level of formality and standardization of the format of the documents, with a corresponding de-emphasis on content. Since everyone is already busy working on their particular narrowly-defined piece of the solution at any given time, no one is available instantly to take a hand-off. Thus, a process based on hand-offs across the boundaries between speciaists leads to a series of work queues containing batches of work awaiting attention. A batch-and-queue workflow tends to drive work into a fixed sequence of steps based on the specialized roles on the team. Very quickly, the process becomes scenic, wet, and wasteful.
Many people talk about streamlining their development process, about lean thinking, and about agile methods. They talk about self-organizing teams, lightweight management processes, and the "whole team" or "project community" concept. They express great enthusiasm for better ways of working and eagerly join in the popular sport of disparaging the waterfall. They talk the talk.
And then, oblivious to the implications, they ask about specialized roles.
The fact they think in this way exposes a deeply-entrenched traditional mindset that prevents them realizing many of the improvements they're hoping for. The end of the game is to attach "agile" and "lean" labels to all the same processes and methods they were using before, and to praise each other's "leadership" for doing so.
But our world - the world of IT - is changing. It isn't changing in the same way as a light bulb changes from off to on. It's changing in overlapping waves. Every organization is not at the same point. Every department, team, project, and individual in a given organization is not at the same point. Those who are working in the midst of the interference pattern created by the overlapping waves of changes may feel overwhelmed by apparent chaos. Sometimes it's hard to know what to do, or what is even feasible to try in a given organization. In this uncertain world, one thing at least is sure: To try and make anachronistic specialized roles fit the new way of things is a recipe for failure. Those who take that approach are likely to have a number of other basic things wrong, too.
So, what are these so-called "waves?" Here's how I see it (year ranges are approximate):
Previous/current wave, circa 2000 - 2009
- Workflow: from linear waterfall to iterative batch-and-queue
- Development approach: from code-and-test to test-driven development
- Customer relationship: from the service provider model to direct customer collaboration
- Management approach: from the Theory X manager to the Theory Y manager
- Engineering mindset: from the narrowly-focused engineer to the business-aware developer
- Career development focus: from the narrow, deep specialist to the cross-discplinary team member
- Organizational structure: from walled and moated fiefdoms to 'aligned' technical and business structures
Current/next wave, circa 2004 - 2013
- Workflow: from iterative batch-and-queue to lean pull and flow
- Development approach: from test-driven development to test-driven requirements
- Customer relationship: from direct customer collaboration to the seamless organization
- Management approach: from the Theory Y manager to the coach/enabler of self-organizing teams
- Engineering mindset: from the business-aware developer to the business-oriented technologist
- Career development focus: from the cross-discplinary team member to the generalizing specialist
- Organizational structure: from 'aligned' technical and business structures to business/technical fusion supported by a core infrastructure unit
And let's not overlook this one: The big-bang transformation, for those who have seen the future and know it works
- Workflow: from linear waterfall to lean pull and flow
- Development approach: from code-and-test to test-driven requirements
- Customer relationship: from the service provider model to the seamless organization
- Management approach: from the Theory X manager to the coach/enabler of self-organizing teams
- Engineering mindset: from the narrowly-focused engineer to the business-oriented technologist
- Career development focus: from the narrow, deep specialist to the generalizing specialistr
- Organizational structure: from walled and moated fiefdoms to business/technical fusion supported by a core infrastructure unit
When it comes to role specialization, career paths are evolving in a couple of directions. The first will be easier for traditionalists to grasp. It's the continuation of the classical technical career path, involving deeper and deeper skills in a chosen area of specialization. Deep specialists will continue to play an important part going forward. The difference is that very few business software development projects will need this sort of specialist on the team. Instead, they will be external resources that may be called in to deal with particularly sticky problems. These are the people who will operate and sustain the core enterprise technical assets of an organization. They are not the ones who will carry out business-oriented application development.
The deep skills of the specialist would be wasted on routine, day to day tasks that can be carried out by the second type of professional in that world, the generalist. And the narrowness of the specialists' skills would make them largely useless on the majority of business-facing development projects in which team members are expected to wear many hats in the course of a typical day, and to be equally effective wearing any of them.
Here's a picture a colleague of mine, Ashley Johnson, likes to draw when explaining Theory of Constraints and lean thinking.
The diagram represents a manufacturing cell. You recognize it from all the books and presentations about lean manufacturing that have been making the rounds the past few years. A customer requests a product - the lean imagery is that the customer "pulls" the product out of Station 6. The input to Station 6 is the output from Station 5, which doesn't produce any output until Station 6 pulls it, so there are no work queues but there is a steady flow of work, all of which contributes to the delivery of customer-defined value. And so it goes, all the way up the line to the beginning. And of course you could scribble some numbers between the boxes to depict the size of the work queues between stations, and go through the basic ToC stuff. All well and good for a manufacturing operation. How does software development map to this model? Something like this:
Here we have the conventional steps of a "waterfall" software development process mapped onto the production cell. This could be a classical linear waterfall or an iterative waterfall like a RUP process. The essence is the same either way.
At this point it isn't a lean process. Here, the customer makes a request and someone or something (probably a lot of someones following a heavyweight project initiation procedure) starts the production process by "pushing" the raw materials (the project charter, vision statement, risk assessment, etc.) into Station 1. Station 1, manned by specialists in the fine art of requirements elaboration, works as fast as possible (that is, their production pace has nothing to do with the ability of the next group of specialists to consume their product) to produce requirements and throws them into a queue where they wait until Station 2 has time to pick them up. And so it goes down the line, each station manned by the appropriate specialists, until eventually something pops out the end of the process. Something. Eventually. Maybe.
What happens when people begin to apply agile development practices? One of the first things people have done in the past few years is to apply test-driven development to the software design and code part of the process using a test-first approach. Today you might consider this the normal way of working, or you might consider it bizarre, freakish, unworkable, unproven nonsense. It just depends on what part of the interference pattern you happen to be living in at the moment. In any case, when this is done a funny thing happens to Stations 3 and 4:
A funny thing happens to some of the traditional roles on the project, too. The former distinction between software designer and programmer is all but erased. Also, a good deal of the basic testing is now handled by the developers themselves, who use unit tests as executable "specifications" to drive (test-driven...get it?) the coding. As a result, QA testers now find it unnecessary to discover all the little bugs programmers inevitably create in the process of developing the code. They can spend more time on value-add testing actitivies like system testing, acceptance testing, performance testing, capacity testing, failover testing, and so forth. The business analyst role changes for the better, as well. BAs now find it unnecessary to write detailed coding specifications. They can spend more time on getting an accurate and complete picture of what the user actually wants.
Once this TDD thing really gets grooved in, another level of metamorphosis occurs among the technical members of the team. Since they now design, code, and test all in one go, they begin to "level up" their skills in all those areas. Another agile practice, pairing, facilitates this levelling-up of skills, especially when people pair across disciplines. People change the general direction of their professional development, de-emphasizing narrow and deep specialized skill development in favor of building broader and more balanced skills. They become what Scott Ambler calls generalizing specialists. There's more of this particular metamorphosis ahead, so keep it in mind. It has a lot to do with why I call the traditional roles "anachronistic."
A lot of organizations are more-or-less at this stage just now. That's why we see so much focus in blogs, discussion boards, user group meetings, professional conferences, and magazine articles on the question of "How does traditional role X change when agile methods are used?" The short-term answer is something like the comments I made a couple of paragraphs ago - some of the burdens shift around a bit, leaving certain specialists with more time to do value-add work. And everyone wants to do value-add work. The longer-term answer is, perhaps, not quite so happy for certain specialists, but we aren't there yet. For now, there's plenty for everyone to celebrate and no obvious career threats to those who can't or won't broaden their skills.
What happens to our little manufacturing cell if we extend the idea of TDD to the functional level? Well, a funny thing happens to stations 2 and 5. Stations 3 and 4, too. (You can see where this is headed, right?)
Now the technical team begins by writing executable functional tests and works from there to create software that makes the tests pass. There's still good old TDD going on, but it's happening in little loops of activity driven by the executable functional tests. So now the group of generalizing specialists expands to encompass a broader range of skills. They have to level up their skills in requirements analysis and QA testing. At this point the specialized QA testers are even happier than before, because there is even less mundane testing work for them to do and they can focus even more strongly on the real value-add testing they can contribute at the full-system level.
Like the QA testers, the Business Analysts are now working much more closely with the technical team members than ever before. They collaborate to create functional specifications in an executable form. This is a tremendous time-saver that avoids many misunderstandings, exposes missing or incomplete requirements very early in the process, and prevents most bugs from being created.
Extending the idea of TDD to the acceptance testing level, and incorporating system-level testing with the normal activities of the development team, leads to even further simplification of the cell:
At this point, it becomes clear that there is no particular need for a separate, specialized Business Analyst role. There is no reason the developers can't talk directly to the customers about requirements and then demonstrate the results to them.
It's worth mentioning that the origin of the Business Analyst role harks back to a time when business people and technical people needed a sort of "translator" to convert biz-speak to tech-speak and vice versa. There were no such things as home computers, cell phones, and PDAs. Cars, TV sets, and toasters were not computerized. Most people were uncomfortable around technology. At the same time, the majority of programmers were heads-down bit-twiddlers who couldn't have cared less about "business value." They were two communities that spoke different languages, had different motivations, and held different worldviews. The BA, a specialist in bridging the communication gap between these two communities, rose to meet the communication challenge.
But a great deal has changed since then. Today's BA is mainly concerned with producing documents for their own sake. The role persists out of organizational inertia; we've had BAs ever since anyone can remember, so they must be necessary. Why else would they even exist? Once an organization gets to the point depicted in diagram 5, there is no longer any way for the BAs to pretend business people need them to communicate with technical folk, and no way for them to pretend technical folk need them to comprehend business people. The technical folk and business people are now face to face. To their mutual surprise and joy, they discover that in this era of ubiquitous information technology, everyone is pretty comfortable talking about computer stuff, and everyone is pretty comfortable talking about business stuff. It's all just stuff.
The QA specialists may be able to argue that they can still add value by performing system-level testing not usually conducted during a software development project. But do we really need a separate group of people to carry out that function? It turns out there is nothing a QA specialist does that a developer can't do just as well.
Lest this sound like nothing more than a rant, let me say that I took Scott Ambler's description of the generalizing specialist to heart when I first read it several years ago. When I was first learning about agile development, I realized that the only way to grok it fully was to work in every role on agile projects; real, enterprise projects. Since then, I have managed to wrangle a real job assignment on at least one project in every role defined by the most popular agile methods with the exception of Customer or Product Owner. I have literally been there and done that, as a BA, as a QA tester, as a project manager, as a ScrumMaster, and as a development generalist who carried out tasks ranging from architecture to design to coding. I have seen the secret knowledge the specialists claim justifies their existence. I have returned from the adventure ready to report to you, dear reader, that none of it is a Big Deal. The only piece of the puzzle that is even somewhat difficult is the programming. This is why I say with confidence that in the emerging IT world of generalists there is no need for entirely separate groups of people to specialize in these relatively non-differentiated skills.
In fact, there is no need for the IT department as such to employ business application developers at all. They can work directly for the business units they support. Nor is there any particular need to run "projects" as such. Customers can simply pull features as they need them, as depicted here:
The IT department can become a truly technical department with the mandate to satisfy needs for technical standardization, economies of scale, consistent service interfaces, enterprise security, system and data recovery, networks, data resources, package integration, and other technical infrastructure issues. But this department need not run "projects" on behalf of business "clients" who are part of the same corporation. The organizational model looks something like this:
The old (current) model puts any and all activities related to "computers" in one department, and any and all activities related to "business" in other departments. In the new world, this sort of segregation is not only meaningless, but counterproductive.
So if you want to know my answer to the question of how the traditional roles will change in an agile world, there it is. Not to worry, though; it won't happen overnight.