I have been working closely with members of IBM's Rational team recently, as they have been helping my employer learn to use the IBM/Rational toolset for software development, project management, software testing, and so forth. This is the familiar Rational Software suite, including Rose, RequisitePro, ClearQuest, FunctionalTester, and all the rest. IBM is currently in the process of integrating the Rational tools with its Eclipse-based development suite, formerly known as the WepSphere Studio Application Developer.
After a long and intense working session, one of the IBMers asked us how our Agile software development practice was coming along. In the course of that discussion, he mentioned that RUP should serve as a very powerful Agile development methodology. I responded that RUP was a Waterfall-oriented methodology, and not really aimed at Agile development.
He and his colleagues were very surprised by that comment. They said other Agile practitioners had told them RUP and XP were "totally different," yet they did not understand why, and the Agile practitioners who told them that were unable to elaborate. It sounded to me as if the Agile practitioners were being defensive about their preferred approach, just as advocates of traditional methods tend to be defensive about theirs.
I don't believe defensiveness can lead us anywhere useful, as a professional community. Instead, we need to understand when and how to apply all the tools at our disposal, including both RUP and XP.
Aren't both RUP and XP examples of iterative methodologies, the IBMers asked, and aren't iterative methodologies part and parcel of Agile development? What could be so terribly different, then, and why could RUP not support Agile development? And what about "control?"
I thought their questions were relevant and deserved more than an off-the-cuff answer. Indeed, if there is confusion generally in the developer community about which methodologies can support each approach to development, then we all need better answers. I've been giving the matter some thought, and this paper is the result.
As with most questions in our field, the first place to look for answers or explanations is in the work of others. It's rare to be the first person to confront any given question or problem. Someone may already have come up with an answer. So, I searched the net.
I searched for a straightforward comparison of RUP and XP, or a case study of how RUP has been adapted to an Agile project, or any other useful description of how RUP may be used for Agile development, and came up short.
The homepage of Scrum, Controlled Chaos, states that Scrum is a process that "wraps" methodologies such as RUP and XP. However, the site contains no clear explanation of what that statement means in practice.
The noted UML expert Sinan Si Alhir has attempted to reconcile RUP with XP, notably in his 2001 paper (no longer available) "XP, the Agile Alliance, and RUP," but he appears to be biased in favor of RUP. His argument is aimed at portraying RUP as "more agile" than XP rather than at clarifying the similarities and differences in the two methodologies in an objective and useful way.
He states, "RUP is a more massive but more agile methodology than XP, from which appropriately weighted and more agile processes than XP may be derived, with which more agile projects than XP may be executed." Frankly, this comes across as a load of gibberish calculated to sound academic and sage. There is no value in trying to elevate one methodology over another. Each has its strengths and each can support projects whose characteristics lend themselves to a particular methodology.
It seems there are many practitioners who are defensive about UML and RUP. It's puzzling, since Agile methods pose no threat to these technologies. The advent of the Agile approach means that UML and RUP may no longer serve as the centerpieces of an organization's entire software development process, but this certainly doesn't rob them of their value. An open-minded professional ought to be interested in how best to use various tools to achieve excellent results, rather than in defending his/her past investment in any particular tool.
Unfortunately, the word "agile" has gotten so much play in the trade press and in informal buzz around IT organizations that a lot of people are using it to describe just about anything that seems in any way flexible or innovative. Of course, that's not exactly what Agile development means. The 2004 book Balancing Agility and Discipline: A Guide for the Perplexed, by Barry Boehm and Richard Turner, offers a pretty good definition of Agile methods:
"In general, agile methods are very lightweight processes that employ short iterative cycles; actively involve users to establish, prioritize, and verify requirements; and rely on tacit knowledge within a team as opposed to documentation. A truly agile method must include all of the following attributes: iterative (several cycles), incremental (not deliver the entire product at once), self-organizing (teams determine the best way to handle work), and emergence (processes, principles, work structures are recognized during the project rather than predetermined). Anything less is a 'lightened defined process,' although such processes can exhibit considerable agility." (page 17).
An experienced software developer can readily see the ways in which RUP is most obviously consistent with some of those attributes: It is an iterative process that delivers results incrementally. So, is RUP a truly agile method, or is it a "lightened defined process?" To find a more comprehensive definition, let's turn to Wikipedia, which offers an accurate and concise overview of RUP.
If you are accustomed to working in an Agile way, the description of RUP on Wikipedia will immediately strike you as highly formal and structured. A RUP iteration consists of four phases that always proceed in sequence - inception, elaboration, construction, and transition. Each phase is completed when defined milestones are achieved, and the milestones are achieved when certain documentation artifacts have been created. In practice, a certain amount of overlap occurs between the phase just ending and the one just beginning, but the phases are still distinct.
A Rational Edge article [pdf] begins, "In contrast to a waterfall approach, an iterative approach like RUP emphasizes..." and so on. This is fairly representative of many articles, papers, and books that compare and contrast RUP with Waterfall methods. There is a widespread assumption that RUP represents an Agile approach, and not a Waterfall approach. Why?
The basic concept of the "waterfall" is that water flows only downhill; never uphill. Once a checkpoint has been reached on a project, there is no going back. Once requirements have been documented and signed off, they can never change; once the design has been documented and signed off, it can never change; and so it goes. The idea behind this is that careful and thorough risk assessment and up-front design will minimize project risks. In practice, we know this is rarely the case in software development work, although it may well be true in other areas of endeavor, such as manufacturing, for example. RUP allows water to flow uphill for limited distances, and under controlled conditions. Therefore, its proponents do not consider it a Waterfall process, strictly speaking.
There is a concept in the RUP world, supported directly by Rational Rose, of bidirectional reengineering. This is a forerunner of the concept of refactoring. Bidirectional reengineering never caught on in a big way because it is inherently error-prone and impractical. Basically, bidirectional reengineering means that source code can be generated from the model, and an updated model can be generated from the source code. The tool depends on developers to adhere to strict rules about where and how to insert custom code into source code that is generated from the model. These comments allow Rose to recognize and preserve the custom code when regenerating a new version of the model and subsequently, after the model has been refined, regenerating a new version of the source code.
Basically, this approach requires that the primary artifact in the development process is the model. The model is expected to be expressed in terms of UML, and comprehensive, formal documentation of the model must be developed and kept up to date as code is built. The set of formal documents would normally include Class and Object Diagrams, Use Case Diagrams, Sequence Diagrams, Collaboration Diagrams, Statechart Diagrams, Activity Diagrams, Component Diagrams, and Deployment Diagrams. Each of these artifacts presents a different view of the solution, concentrates on specific aspects of the solution, and is intended for a different audience. (See Sinan Si Alhir's UML in a Nutshell, or a similar book, for more information.) In practice, not every one of these diagrams is necessarily produced for every part of a solution, but the set of diagrams remains central to RUP development.
In an Agile project, the primary artifact in the development process is the code. It is contrary to Agile principles to go back and update model diagrams once they have been used to start code development. Once code exists, it becomes the primary artifact. It is considered counterproductive to keep the model diagrams up to date at all stages of development. If there is value in retaining a final version of comprehensive UML diagrams to describe the solution, it is best to produce those documents based on the version of code that is delivered rather than to worry about incrementally updating the diagrams day by day. Therefore, any process in which the primary development artifact is the model cannot really support Agile principles to the fullest extent. It imposes a certain amount of documentation overhead on the project team, above and beyond the level of documentation that contributes to building the solution.
Even though there is no concept of water flowing downhill in the Agile approach, there is a very definite concept of moving foward toward the delivery of a solution. Is it necessary to update the model every time the code changes? Maybe yes and maybe no. If it helps move the project toward its goal, then the answer is yes; otherwise, the answer is no. The end user will never see the model; therefore, it is not a "product" in its own right.
Scott Ambler, in his 2002 book Agile Modeling: Effective Practices for eXtreme Programming and the Unified Process, provides some practical guidelines for getting value from RUP artifacts without becoming mired in process-related documentation. He writes, "[Agile Modeling] addresses design modeling and refactoring addresses design improvement of source code. This begs the question, 'What do you do when you have an existing design model and you refactor your code?' Although it's an interesting question, the real issue is that you have two artifacts, a design model and source code, that describe the design of your system. The source code has changed; now you need to decide whether or not you wish to update the model. [...] How do you apply AM and refactoring together? Apply AM practices as appropriate when you are modeling, use those models as input into your programming efforts, and refactor your code as you normally would have." [pp. 187-188]. Note that there is no requirement to keep the model in sync with the source code. You only need to keep the model up to date if you need the model to make progress with development. Otherwise, it is merely ceremonial overhead.
Each RUP iteration looks very much like a miniature Waterfall process from start to finish. That is exactly why I see RUP as a Waterfall methodology. It brings the advantages of iterative development and incremental delivery to a project, but it does not really deviate from the Waterfall model, which calls for risks to be identified in advance, the solution design to be finalized in advance, and (usually) testing to occur after the fact.
To the traditional thinker, this is the only way to "control" a project. One must plan thoroughly in advance, and "manage" change. The traditional thinker sees Agile methods as inherently ad-hoc and uncontrolled.
But think about what that mindset implies. The very idea of "managing" change implies that change is an anomoly that must be handled outside the normal scheme of things. The Agile approach embraces change as the norm, rather than treating it as an anomaly. Methods that work well with Agile development handle change in the normal course of things rather than as an additional task. That is why Agile methods may not appear to offer "control" until one gets into the flow of an Agile project. The control of change is built right into the everyday work of the project team. It simply doesn't require a separate set of processes for "change control." Change is a reality in software projects; the Agile approach accepts that reality and works with it naturally.
RUP includes the definition of various project management processes, specifies particular sets of documentation that pertain to each milestone, and requires certain best practices that might be a little too low-level for a methodology to define. For example, it prescribes a component-based solution, as if that approach were a one-size-fits-all way to build software; it prescribes UML as "the" language for specifying design; it prescribes Use Cases as "the" way to specify requirements; and so on.
In an Agile project, many of these implementation details would be "discovered" as the team incrementally builds the solution. The prescriptive nature of RUP is, therefore, one of the fundamental differences between it and an Agile-oriented, adaptive methodology such as XP.
It is true that because it supports iterative development and incremental delivery, RUP alleviates some of the problems inherent in the predictive approach. However, it stops well short of supporting all aspects of Agile development as envisioned by the Agile Alliance, and as practiced by Agile development teams.
One of the key principles of Agile development, and one of the reasons Agile teams often bring in results much faster than traditional teams, is that any given team is assigned to exactly one project at a time, and that team follows the project through from inception to production.
Looking at the standard set of RUP phases, an Agile practitioner must ask, what are the developers doing during the inception, elaboration, and transition phases? A smart-aleck will reply that the developers are testing their code; although RUP prescribes most other aspects of development, it does not prescribe test-driven development. The more objective answer is that in traditional IT shops, individual developers are typically assigned to more than one project at a time. During the inception and elaboration phases of one project, they are busy writing code in the construction phase of another. This sort of discontinuity is contrary to the Agile approach, and illustrates another way in which RUP isn't exactly agile.
The roles in RUP are derived from the professional specializations that have become typical in large IT organizations over many years. There is a more granular division of labor than is the case with the Agile approach. Specialists hand off work to one another at specific points in the process, supported by a substantial amount of formal documentation. Each group of specialists depends on the quality of the documentation to guide their work. For that reason, the roles in RUP do not map directly to those of an Agile methodology such as Crystal, Scrum, or XP. Agile practitioners do have their individual areas of specialization, but not to the exclusion of other work; they tend to be generalists rather than specialists.
Although the IBM/Rational toolset is not yet fully integrated, it is clear that IBM is following the same direction with the tools as Rational Software had done. Each is designed to support the work of a specialist working in one of the roles defined in RUP. This indicates the guiding mindset behind the project is based on Waterfall thinking. The concepts and approach underlying the methodology and the tools that support it are non-Agile. That is not "wrong" or "bad" in itself; both the concepts and the supporting tools are well suited to Waterfall projects. The error is in assuming they are equally well suited to Agile projects.
Another basic tenet of Agile methods is test-driven development. Developers write executable test cases either before writing a unit of code, or immediately afterward. Unit tests and integration tests are automated so that with every code check-in the complete suite of tests is run, and developers are notified immediately of any problems. By the time they complete an iteration, every unit and integration test passes.
RUP does not require test-driven development, although it does require a "test soon" approach to unit testing. The fact RUP does not require test-driven development becomes very clear when one assesses the capabilities of the remainder of the IBM/Rational suite. FunctionalTester has no "mock" feature; it is not possible to develop executable tests that reflect the requirements, and then use the tool to verify that the code fulfills those requirements. It is only possible to develop executable tests that mimic the actual behavior of the code. This is yet another way in which RUP is not truly an Agile methodology.
Agile methods are based on the notion of "aggressive refactoring." Tools that support refactoring are critical to Agile projects, since the design and structure of the solution are subject to significant changes from one iteration to the next. Refactoring is the practice that most strongly supports the "discovery" of the final design; a fundamental Agile concept.
In contrast, RUP is entirely predictive with respect to solution design. UML is to be used to elaborate Use Cases and other design artifacts, which in turn become the basis for construction. All of this is traditional Waterfall development to a T.
One of the things that makes Agile agile is keeping iterations short. Customers need to see incremental results frequently, and they need to participate in IPMs frequently, so that the project doesn't exceed their attention span. Agile iterations, regardless of the methodology used on the project, tend to be about two weeks long.
Because of the higher ceremony, strict division into phases, and required documentation artifacts in RUP, at iteration length of two weeks is simply infeasible. One to three months is more typical. With iterations of that length, a project can easily lose momentum, and stakeholders can lose enthusiasm and interest as they become distracted by other responsibilities and events. This factor tends to place RUP more the Waterfall category rather than the Agile category.
Controlled Chaos claims that Scrum wraps RUP, as well as methodologies such as XP, in an Agile process. As a practitioner who has worked in both worlds, I have to say I am unconvinced of that. To wrap RUP, one would have to modify RUP considerably to make it fit into the general Agile model.
For one thing, several RUP roles are really the same role in Agile (for instance, architect, designer, coder, and tester). This is necessary to keep the work continuous; in a pure RUP project, developers have no work to do while they wait for the elaboration phase to finish.
For another, many of the formal documentation requirements and ceremonial milestones would have to be relaxed in order to keep the iterations short enough to be productive. In short, by the time RUP was fully wrapped, it wouldn't be RUP anymore; it would be a variant of XP or Crystal.
To a large extent, RUP depends on UML, and many of the artifacts one must produce to satisfy milestone requirements are UML diagrams of one kind or another. Does this fit into an Agile development process? I believe it does, but not exactly the same sense as RUP aficionados do.
In a RUP project, Use Cases would be the primary means of elaborating requirements. I have found Use Cases to be a valuable tool on Agile projects, as well. However, I do not believe Use Cases can fully elaborate requirements in a way that fits smoothly with Agile practices.
I see Use Cases as valuable in an Agile project for three purposes: (a) as a starting point for story narratives; (b) as a basis for documenting acceptance criteria for functional tests; and (c) as a way to identify system interfaces that might be overlooked in other views of the solution.
Sequence diagrams and other UML artifacts may be useful from time to time to clarify some particular aspect of the problem to be solved or the solution under development. While RUP makes UML artifacts central to a project, I see them as tools to be bent to the needs of the Agile development team, and not as deliverables in themselves or even as artifacts necessary for success.
The main difference between the way a RUP enthusiast would apply the process and the way an Agile practitioner would apply it is that the former sees RUP as the core of the entire project methodology, while the latter sees particular bits and pieces of RUP as potentially useful on a case by case basis. In considering the strengths of RUP in the context of Agile development principles, one of the key factors Ambler notes is the importance of clear communication on an Agile project. Since Agile development depends on direct communication rather than formal documentation, anything that interferes with clear communication is a hindrance.
He writes, "Simple tools - including whiteboards, sticky notes, flip charts, and index cards - are easy to work with and are both flexible and non-threatening. As a result, they are likely to be used in team situations, because there is little opportunity to embarrass yourself by revealing you are not adept with the tool. More complicated tools often prove to be barriers to communication, particularly those CASE tools that are single-user because one person is separated from the conversation when using the tool. [p. 85] ...the principle use the simplest tool applies... [p. 87]."
RUP and XP are superficially similar in that both are iterative methodologies and both are aimed at incremental delivery of solutions, with significant participation by all stakeholders throughout the development process. However, on closer inspection, it is clear that RUP is intended to support iterative development within the framework of a traditional Waterfall process, while XP is intended to support Agile development.
The best way to get value from bits and pieces of RUP is to keep design practices separated from coding practices so that the relatively high ceremony of RUP does not get in the way of productive development work. Use the bits of RUP that make sense for your project, but beware of turning toward a process-oriented mindset, and away from an agile mindset.