Turning and turning in the widening gyre
The falcon cannot hear the falconer;
Things fall apart; the centre cannot hold;
Mere anarchy is loosed upon the world,
The blood-dimmed tide is loosed, and everywhere
The ceremony of innocence is drowned;
The best lack all convictions, while the worst
Are full of passionate intensity.
Surely some revelation is at hand;
Surely the Second Coming is at hand.
The Second Coming! Hardly are those words out
When a vast image out of Spiritus Mundi
Troubles my sight: somewhere in sands of the desert
A gaze blank and pitiless as the sun,
Is moving its slow thighs, while all around it
Reel shadows of the indignant desert birds.
The darkness drops again; but now I know
That twenty centuries of stony sleep
Were vexed to nightmare by a rocking cradle,
And what rough beast, its hour come round at last,
Slouches towards Bethlehem to be born?
The Second Coming, W.B. Yeats
People over process
Lean and agile development are often lumped together as if they were one and the same thing. They do have a great deal in common, and in the large both represent alternatives to the traditional methods that have led to the well-documented low success rate of IT projects in the past 25 years. But I see an important difference between the two, and I think the failure to understand that difference is at the root of the problems many organizations are experiencing in their application of agile and lean methods.
I often say the difference between lean and agile development is cultural, and not mechanical. That is, it is almost all about mindset and only a little about following a certain set of practices in cookbook fashion. The distinguishing factor is the degree to which the agile principle of people over process is genuinely embraced by the team, the customer, and the organization. An agile approach emphasizes people over process; a lean approach seeks to remove as much unproductive administrative overhead as possible, but does not change the traditional process-centric mentality.
The agile philosophy is, basically, to place process in the service of people. Qualified and motivated people know what to do and will gladly do it provided they "own" the problem. They need processes that help them get things done, but that do not hinder them in acting autonomously. The lean philosophy, in contrast, is to make the process better so that people can follow it to success. In contrast to a traditional "heavyweight" or "high-ceremony" process, a lean process focuses on getting the job done and dispenses with steps that only add administrative overhead. The belief is that a well-designed lean process can lead "anyone" to success. All they need do is follow the steps the process defines for them, without deviation.
I have been told this is such a fine distinction as to be meaningless for practical purposes, and it amounts to an academic semantical rant. I disagree, because the cultural mindset about people and process leads to a great many small decisions and behaviors throughout the day that affect the success of projects. Understanding, internalizing, and actively living the agile principle of people over process is a critical success factor for agile organizations. It just doesn't get any more practical than that.
It is not by accident that the idea of people over process is stated as the very first principle in the Agile Manifesto 
. When that principle is fully and sincerely embraced, an organization is 90% of the way toward a successful agile culture. When the principle is not fully and sincerely embraced, it makes little difference how much attention the organization pays to the remaining three agile principles or how rigorously they adhere to agile or lean methodologies and best practices; they will naturally look to a process to guide their work.
Many people say all the "right" agile buzzwords, but behave as if, in their heart of hearts, they still revere process as king. The problem is you cannot elevate people above the king and yet bow unconditionally to the king in all things. Paying lip service to the one while doing the other is a recipe for failure; you will never see what is going wrong because you are unaware of your own subservience to process. When a traditional thinker adopts agile practices, he is replacing an old process with a new one; "The king is dead — long live the king!" When a non-traditional thinker adopts agile practices, he is reversing the historical emphasis on process over people; "The king is dead — viva democracy!"
When people slouch back towards their old habits unknowingly, they are doing neither agile nor lean development — at least, not intentionally. Since they are not really doing what they believe they are doing, their results are likely to be less than optimal and the reasons are likely to be less than obvious to them. Either lean or agile development will work, but not by accident and not when various practices from each camp are conflated in a manner dictated by a defined process rather than purposefully chosen and guided by fundamental principles.
What rough beast?
In the attempt to embrace agile methods, many organizations fail to make a genuine cultural change to elevate people above process. They approach agile methods in the same way as they have approached every other "new" methodology over the years — as just another cookbook recipe to memorize. They chant the correct mantras, but remain unenlightened. People work in the service of the process, when instead the process should facilitate the work of people. Believing they are doing the right things, turning and turning in the widening gyre, they cannot hear the falconer as he calls out to them the first principle of people over process.
What they end up with is a familiar beast, after all. It is familiar enough that Certified Scrum Master instructors present the resulting process model as a counterexample to be wary of; a "process smell" to be alert to. It is familiar enough that it is deemed an anti-pattern , and has a name: Staggered Iterative Waterfall. The illustration below shows how agile consultant Kane Mar depicts this anti-pattern:
At first glance, this process looks pretty efficient. People in various specialized disciplines are working simultaneously on assorted tasks that all contribute toward reaching a common goal. So, what is the problem? Why is this an anti-pattern?
The first problem is that all the team members are not working on the same iteration. While developers write code for iteration n, analysts define requirements for iteration n+1 and testers validate the functionality of code delivered in iteration n-1. You might ask, why not have everyone busy doing something useful? I might answer, why not indeed; but this is not useful. Analysts have no way to benefit from the lessons being learned by developers. Testing activity is deferred longer than necessary, delaying the discovery of bugs that need to be fixed. Team members are not collaborating across disciplines. They cannot discover the new perspectives on the solution that would emerge through such collaboration; nor can they become generalizing specialists , thus enhancing the organization's capability to deliver high-value solutions.
The most powerful way in which iterative development delivers superior results is by enabling continuous learning throughout the project. An iterative waterfall process achieves this by providing a feedback loop from the Construction and Transition phases of iteration n to the Inception and Elaboration phases of iteration n+1. When the Inception and Elaboration phases of iteration n+1 are carried out simultaneously with the Construction and Transition phases of iteration n, the feedback loop is broken. Lessons learned in iteration n do not easily flow into iteration n+1 because the Inception and Elaboration phases are finished before the lessons from the previous iteration's Construction and Transition phases can be transmitted.
An agile process supports continuous learning because all team members across all professional disciplines collaborate directly at all times. Pairing across disciplines transmits new information immediately. Test-driven development and aggressive refactoring keep the solution fully up to date with the latest understanding of requirements and pay off design debt as it is incurred. The open communication and tight, continuous collaboration that characterize agile development are destroyed by the role specialization, formal hand-offs, official documentation, and undermining of the sense of collective ownership that the Staggered Iterative Waterfall anti-pattern imposes.
When the Staggered Iterative Waterfall is used on a project the team thinks is agile, there is another, more insidious problem — different team members depend on different forms of communication as the definitive way to convey information about the solution. Those who genuinely think in an agile way regard direct, face-to-face conversation as the definitive communication channel, and see documented requirements as secondary. Meanwhile, those who think in a traditional way regard the documented requirements as definitive, and the face-to-face discussions as helpful but "unofficial" unless and until the results of those conversations are codified in the formal documents.
The agilists on the team see the requirements documents as transient artifacts whose usefulness extends only as far as they help developers produce working code (see the Agile Manifesto for a comment about the primacy of "working code"). The traditionalists see anything that has been formalized in a document as "official" and anything else as "unofficial." They see the requirements documents as deliverables in their own right, as important as the working code itself. This difference in mindset stems directly from organizational culture.
Things fall apart; the centre cannot hold;
Mere anarchy is loosed upon the world...
Ironically, this anti-pattern is rarely recognized as a problem and even more rarely fixed. Why? Because despite its faults, it yields better results than the traditional, linear waterfall approach. But those who are interested in enjoying the benefits of real agile or lean development should not be satisfied just with that. They can achieve much more by choosing an approach with purpose and then following it with discipline.
Agile development (Scrum, XP)
Agile development is based on a few simple principles including the guidelines stated in the Agile Manifesto, iterative development with incremental delivery, transparency, and accountability. The most mature and widely-used process for agile work is called Scrum . It is a lightweight management wrapper that supports empirical process control. While not limited to agile methods or to software development projects, Scrum is the most natural fit for agile software development projects. The general Scrum cycle is shown in the illustration below, from Mountain Goat Software:
At the more detailed level of development methodology, the most widely-used for agile projects is called eXtreme Programming , or XP. It complements Scrum nicely, and defines a set of specific development practices to maximize the effectiveness of agile teams. One of its core practices, pairing, calls for all work to be done by two people working together rather than by individuals working separately. This improves productivity in a number of ways and also facilitates cross-disciplinary collaboration in the closest possible way. Other practices tighten the level of collaboration in various ways — test-driven development blurs the role distinctions of analysts, developers, and testers; daily customer participation blurs the role distinctions of analysts and developers; refactoring and design discovery blur the role distinctions of technical disciplines on the team, and so forth.
Iterative development and the iterative waterfall (RUP)
Iterative development offers significant advantages over traditional, linear development because it has proven to be impossible to understand all risks, costs, implementation details, and future user preferences in advance and in detail through big design up front . It is also well known that there is a point of diminishing returns in estimating whereby the accuracy of estimates falls off sharply when one puts "too much" effort into refining an estimate in advance. Our understanding of these details must be improved over time through incremental delivery of subsets of a solution. This is really the only practical way to build a high-quality solution.
The most mature and widely-used iterative methodology is the Rational Unified Process , or RUP. The four phases of a RUP iteration look like a miniature waterfall: Inception, Elaboration, Construction, and Transition. The work of each phase tends to fall mainly to a single professional discipline. In most organizations, a RUP iteration is treated in much the same way as a short waterfall project, with formal hand-offs of interim deliverables, mostly in the form of standardized documents, from one set of specialists to the next through some sort of "high-ceremony" procedure. Effectively, this amounts to an iterative waterfall process.
An iterative waterfall process is easier for traditionalists to grasp and more comfortable for them to work with than a "pure" agile process. They gain the benefits of iterative development and incremental delivery without surrendering the illusion of control provided by "heavyweight" documentation artifacts and formal meetings, checkpoints, quality gates and the like. Because the team is splintered into different specialized disciplines, there is no sense of collective ownership and no accountability. In addition, team members can stick to their familiar professional disciplines without the "pressure" of trying to become generalizing specialists and the implied "threat" this makes them feel to their "job security."
Although RUP is most often used to support an iterative waterfall process, its creators envisioned it as a more agile methodology than that. One of RUP's leading advocates, Sinan Si Alhir, explains this in an article entitled Understanding the Unified Process (UP) . Where he writes of the sides of the parallelogram collapsing into a diagonal line, he describes the way in which RUP is typically applied.
An iterative approach involves using a mixture of sequential and linear approaches where linear approaches focus on the problem and sequential approaches focus on the solution. Figure 5 shows the overall pattern of how effort is distributed using an iterative approach, resulting in a parallelogram shape where the corners of the parallelogram are adjusted based on the specific project. When all of the sides of the parallelogram "collapse" into a diagonal line, a "pure waterfall" approach results with disciplines are distributed across iterations.
An iterative approach focuses on a stepwise refinement of knowledge throughout the lifecycle. During the Inception phase, linear approaches focus on scope and sequential approaches focus on an architectural proof-of-concept. During the Elaboration phase, linear approaches gain architectural coverage and sequential approaches focus on addressing architectural risk. During the Construction phase, sequential approaches promote deployment opportunities. During the Transition phase, linear and sequential approaches focus on system completion and project closure.
Generally, an effort ramps up at the start of a development cycle, reaches an optimum where all core disciplines are being performed in parallel and all supporting disciplines are operating as appropriate, and then ramps down at the end of the development cycle. As iterations execute, their content involves collaborations among roles, activities, and artifacts where activities are related via a producer-consumer relationship and may overlap in time such that a consumer activity may start as soon as its inputs from producer activities are sufficiently mature.
Alhir's diagram shows the entire team working collaboratively, across disciplines, on deliverables for the same iteration. Since RUP can support this, an obvious question is, Why do people tend not to use RUP in a more agile way? Why do they limit themselves to a straight iterative waterfall? The answer is that because the agile style of work has all team members collaborating directly on all the stories for the current iteration, even across the "boundaries" of professional disciplines, the need for the four RUP phases becomes questionable. If there are no hand-offs, no checkpoints, no quality gates, and no "permanent" formal documents, then what exactly would one do at the transition points between phases? In practice, the phases simply evaporate and the process simplifies itself into something along the lines of Scrum. The reason RUP is used for iterative waterfall processes and not for agile processes is simply that it is unnecessary in a true agile process; its four phases become unproductive administrative overhead.
The darkness drops again...
If agile is a legitimate and effective way to do projects, and iterative waterfall is a legitimate and effective way to do projects, then how is it that so many organizations slip into the staggered iterative waterfall anti-pattern? Why would people go out of their way to create a process guaranteed to be less effective than well-known alternatives? It comes back to my earlier point that the distinction between agile and lean development — indeed, the difference between agile and non-agile development of any flavor — is cultural rather than mechanical. It is a shift in mindset and not merely following some defined methodology or process by rote. Some of the characteristics of agile development are very challenging cultural changes in many organizations, especially transparency, accountability, and collective ownership.
In part, the problem boils down to a simple question of exposure. Working on a task outside one's usual area of expertise, and doing it while pairing with a colleague who specializes in that area, makes one's level of knowledge inescapably apparent to the entire team. For many people it is far less stressful to perform familiar tasks repetitively than to take the "risk" of showing how little they actually know about matters outside their particular professional specialty.
Another issue is the fact that some of the basic ideas and practices of agile development are still relatively new in most organizations. Not all developers really understand the value of key practices that generate high value, such as pair programming and test-driven development. They treat these practices as mechanical tasks dictated by a methodology such as XP. When the pressure is on — a project is behind schedule, the customer is demanding results, and a deployment date looms near — the tendency is to set aside some of the most important agile practices in the interest of speed. They misunderstand the fact that speed and quality result from the disciplined application of agile practices; the practices are not merely "decorations" of some popular process or "new buzzwords" that describe the same old practices as before. This lack of understanding leads people to slip back into old, familiar ways. They may not even realize they have slipped back into anything, since they do not truly understand the difference.
It is not only a question of individual habit or conditioning, though. Many managers have a significant investment in traditional methods, in terms of their own credibility and status in the organization. They know how to facilitate meetings, create Gantt charts, and avoid accountability in ways that assure they will appear competent, concerned, and productive, and that guarantee them a predictable upward career path through the organizational hierarchy. Agile methods offer a way to achieve superior results with fewer people, less time, lower costs, and a lighter management touch than traditional methods. Through common sense, transparency, and a relentless focus on quantifiable value, agile methods represent a real or perceived threat to a certain type of person common in many corporations and bureaucratic organizations. They will do whatever they can to prevent any genuine change, even as they sing the song and dance the dance.
Another type of resistance may be found at the executive level in IT departments. Many CIOs and CTOs follow a career path that consists of moving from company to company and implementing organizational changes that result in short- and medium-term improvements to the bottom line at the cost of long-term benefit to the organization. They move on before the long-term costs kick in. Agile methods make this pattern unsustainable by making cause and effect visible to everyone at all levels. It is possible to replace a pure-overhead operation with another pure-overhead operation only when organizational processes enable obfuscation. In an organization that practices transparency, the fact an operation is purely overhead becomes obvious very quickly. This particular CIO/CTO career strategy cannot survive in a transparent environment. If an organization has this type of CIO or CTO, they will undermine any effort to introduce agile methods.
Most large organizations are highly political, and nothing is more dangerous in a political environment than to take accountability for anything. An elaborate set of hand-offs based on dense, formal documents offers a convenient screen for one's mistakes that can be very hard to penetrate...if anyone is willing to try, in view of the "secrets" they themselves need to protect. Transparency and accountability are very dangerous in many organizations. Errors in budgetary decisions, risk assessment, or project planning can be disastrous if made visible in an organization that thrives on blame-shifting and political maneuvering.
Courage is often listed among the critical success factors for agile organizations. Since agile methods are based on common sense, transparency, and value, it may not be clear why courage is so important. Lack of deep understanding of the value of agile practices, the power of deeply-rooted habits and assumptions, and the risk of political ramifications are dragons that must be slain or tamed if agile methods are to take root in an organization. When the dragons prevail, they push people back to the "safe" territory or traditional methods.
Slouching towards waterfall
An organization can get to the staggered iterative waterfall anti-pattern starting from an iterative waterfall model or an agile model. Starting with an iterative waterfall process, the temptation is to "save time" by folding the universe to create a wormhole connecting three iterations, with developers writing code for iteration n, analysts writing requirements documents for iteration n+1, and testers testing the code for iteration n-1.
Starting from an agile process, an organization can slip back to the waterfall bit by bit if its members do not honestly live the values of agile development. Fear of exposing weak skill sets leads people to retreat to the comfort of familiar tasks, and role specializations are maintained. Role specialization begets hand-offs, which quickly become ceremonial and a "natural" part of the process. Hand-offs beget documentation artifacts, since the specialists do not collaborate directly across disciplines. Documentation artifacts become formalized and start to be seen as deliverables in their own right, at first of equal standing with the working code and eventually as formal prerequisites to development.
And that rough beast, its hour come round at last, slouches towards Bethlehem to be born.