Topic: Lean and agile
When IT professionals learn the basics of lean thinking, they quickly see applications of the ideas to software development. Yet a particular question comes up time and again when discussing how to apply lean thinking to IT: How does the lean concept of inventory map to a software development process?
In Lean parlance, inventory refers to unfinished product or parts that are waiting to be worked on at the next workstation on the line. Understanding how this relates to software development is critical for us to be able to see opportunities for process improvement in our work.
Figure 1 shows a production line with six workstations. Like most production lines, this one uses a batch-and-queue process to produce its product. Each workstation works as fast as it can, individually, to produce as many units as possible of its particular piece of the product.
When individual stations try to maximize their own productivity without considering the larger context of the entire operation, it is called local optimization. People tend to do it because they want to get as much use as they can out of their expensive manufacturing machines. It seems obvious to many people that the best way to accomplish that goal is to keep all the machines running at 100% of capacity at all times. As a result, the stations that have higher capacity generate inventory, since lower-capacity stations down the line cannot process the unfinished pieces as rapidly as they are produced. The Theory of Constraints Toc explains why this is wrong.
To understand how inventory contributes to the problem we need to review the concept of value in Lean thinking. Every process that results in a product or service has a single customer. It may have many stakeholders, but it has one and only one customer. It is the customer, and no one else, who defines "value." Anything in the process that contribues directly to the production of customer-defined value is deemed "good." Everything else is deemed "waste."
That means everything else, including the things that secondary stakeholders consider important, like audit trail information, regulatory reporting information, security...anything and everything other than customer-defined value. The Lean community uses a word borrowed from Japanese, muda, to describe all forms of waste. Why not just call it "waste?" Well, muda makes a stronger statement. If Waste had a mother, the word muda would identify it as "waste" while also insulting its mother.
Some forms of muda are easier to eliminate than others. Some wasteful activities can simply be discontinued; elimimated from the process with no negative impact. We call this Type 2 muda. Of course, there may be other considerations besides an objective assessment of where value is added — the odd management fiefdom nestled deep within a large corporation, for instance, that doesn't contribute to customer-defined value but is difficult to eliminate for (ahem) other reasons ;-) — but that is out of scope for this article.
Other issues may always be with us. Regulatory reporting requirements, auditability requirements, security requirements, and possibly other needs have to be satisfied, even though they have nothing to do with delivering customer-defined value. Because they don't contribute to the delivery of customer-defined value, they are muda. Because we can't just stop doing them, they are Type 1 muda. But muda is still muda, no matter how powerful or insistent the stakeholder may be who wants it. The customer, and only the customer, defines "value."
So, what does all this have to do with inventory?
What does the Customer consider valuable in the process depicted in Figure 1? It's a tricky question, so let me offer you a small hint. There are two words in Figure 1 that describe different sorts of work products generated at different stages of the process. One of the words is "inventory." The other is displayed in a bright, cheery color, a larger font, in boldface italic type, is followed by an exclamation mark, and is located close to the picture of the Customer in the graphic. One of the words describes a sort of work product that has no value whatsoever to the customer, and the other describes the only sort of work product that has any value at all to the customer. Which is which?
Ah, I see you answered that the product has value to the customer, while the various work-in-process inventories have absolutely none. Very good! The product contains 100% of the customer-defined value produced in this process. All the unfinished inventory in the world adds up to a whopping 0% of customer-defined value.
The question of unfinished inventory reminds me of a 9th-grade school report I wrote on the subject of the human digestive system. I wrote it in the first person from the point of view of a morsel of food that described its own journey through the body and its own transformation at each stage in the digestive process. It concluded in just the way you would expect in a paper written by a teenage boy. It never occurred to me that the same model would serve me professionally, hundreds of years later. And yet, here we are. To see how inventory qualifies as muda, it's helpful to imagine yourself as a piece of raw material making its way through a production process.
Your first stop is Station 1. Things happen to you there, and you are transformed in some way. While this is happening to you, you are being brought closer to a state in which you deliver customer-defined value. This is your reason for existence. It makes you feel happy. But when Station 1 is finished with you, you find yourself dumped into a bin along with hundreds, thousands, or even more partially-finished units just like yourself. There, you wait until Station 2 is ready to work with you. During the time you are waiting in the bin, no customer-defined value is being added to you. You're just sitting there, taking up space and time. It is muda. It makes you feel sad.
The pattern is repeated throughout the production process. Even when you are finished, you may sit in inventory for an arbitrary length of time before a customer buys you. Indeed, it is possible that no customer will ever buy you. This is because you were produced before anyone placed an order for you, in hopes that someone would want you someday. It turns out that nearly all your lifespan is spent in inventory.
Local optimization and batch-and-queue processes that generate batches of unfinished inventory are not unique to manufacturing. They are very common in software development, too. Try imagining yourself as a "requirement" as you pass through the software development process depicted in Figure 2.
First, you are put into an inventory of high-level requirements where you await the attention of Business Analysts. While you are waiting, you are not being brought any closer to delivering customer-defined value. You're just sitting there. Eventually, a Business Analysts takes you from the queue and transforms you into specifications of some form. During this transformation, value is being added to you, and you feel happy. But then you find yourself in yet another queue; another bin of unfinished inventory. Now you are waiting for a Developer to work on you. Eventually a Developer takes you from the queue and transforms you into code and, possibly, some form of design documentation. During this transformation, value is being added to you, and you feel happy. But then you find yourself in yet another queue; another bin of unfinished inventory. Now you are waiting for a Tester to work on you. Eventually a Tester takes you from the queue and performs work on you that brings you closer to delivering customer-defined value. Again, you experience happiness. But then you are in inventory again, waiting for a Customer Proxy to do the final acceptance testing on you.
And so it goes, until the end of the process. Clearly, no customer-defined value is added while work waits in inventory. There is no doubt that inventory is muda. But is it Type 1 or Type 2 muda? Is unfinished inventory required for security purposes? To satisfy auditors? To fulfill regulatory requirements? Is it, in fact, of any use to any of the secondary stakeholders associated with the project? No. It is pure waste — Type 2 muda all the way. All that matters to the customer, who is the only definer of "value," is the end product.
You will recognize this idea in one of the four basic statements of the Agile Manifesto: We value working software over comprehensive documentation. In a software development process, inventory takes the form of comprehensive documentation; documentation that will not be used once the product has been delivered. It is interim, unfinished work, whose only purpose is to serve as input to the next station on the production line. The end product of a software development process is, obviously, software. Therefore, software is the only work product that has any value. Everything else is waste.
Many people assume that iterative development solves the problem of waste in software development. If they fail to grasp the importance of inventory as muda, they are overlooking a key part of the solution. Iterative methods based on the Unified Process tend to fall into an iterative waterfall pattern. Within each iteration, the batch-and-queue model is repeated, complete with its hand-offs and interim inventory. Once we recognize the role inventory plays in creating needless waste, we see that simply changing from one big batch-and-queue process to a series of smaller ones isn't going to do the trick.
Figure 3 shows a relatively easy first step toward eliminating inventory in a software development process. Test-driven development replaces the previous two-step sequence of coding followed by unit testing. The key difference is that "unit testing" is not done for the purpose of validation, but rather as a software design technique. The same purpose is served — and probably served better than before, with fewer defects — but with one less batch of inventory — just that much less muda.
Can we eliminate more inventory? If test-driven development makes sense at the unit level, why not apply the same principle at the functional test level? If development is driven by executable specifications, then why do we need batches of inventory between Requirements and TDD, or between TDD and Functional Test? It turns out we don't need that inventory. The key is to apply the idea of the generalizing specialist to the specialized skills of Business Analysts and Testers. Then we can combine the Specifications and Functional Test boxes in Figure 3. If we do this in the context of an agile-style team workspace where team members collaborate directly, it is also possible to collapse the hand-offs between Specifications and TDD, and TDD and Functional Test.
With those improvements in place, we can change the level of detail of the initial Requirements so that the project initiation phase results in a Feature List or Product Backlog, the natural way to feed a Requirements-Driven Development or Storytest-Driven Development process. Finally, we reduce the scope of the Acceptance Test phase so that all it contains is manual user acceptance testing. The majority of acceptance testing can be automated and integrated with the Storytest-Driven Development process. That leaves us with process like that shown in Figure 4.
We've reduced the number of work-in-process inventory queues from 5 to 2; a considerable improvement, and without the need for any deep structural changes. Virtually any IT organization can take the improvement to this level. Once they recognize the importance of inventory as a form of Type 2 muda, the rest falls into place naturally. People start to wonder why they hadn't thought of this before!
But what about those two remaining inventories? We know inventory is Type 2 muda by definition — there are no regulatory, security-related, governance, or business reasons for it. Inventory results from local optimization of stations on the production line that have different capacities, and by the traditional assumption that each station must operate at full capacity at all times in order to be cost-effective. In other words, work-in-process inventory is merely a side-effect of a couple of basic misunderstandings. Okay — so how do we eliminate the remaining inventories?
It requires a bit more of an organizational change than the other improvements we've achieved. The cause of these remaining inventories is the notion of a software development project as a distinct entity that has a start date, an end date, and a team that is formed specifically to carry out the project and is then disbanded, leaving production support to someone else. If we could eliminate software development projects, the last two inventories would disappear.
Now I can hear you saying, okay Dave, I was with you up to that point. But eliminate projects? Are you crazy? How else would we package and prioritize business needs for software, and how else would we plan the work and deliver the results?
The answer comes from two of the basic concepts of Lean manufacturing: Single-piece flow and customer pull. We assume the only way to structure software development work and deliver the results is through projects simply because that's the way it's always been done. What if each business unit had its own application development team? What if the team's mission was not to carry out an individual project but to support a particular business unit? What if the business unit manager asked that team to produce software that supported one new feature or significant modification at a time? What if the team focused on building and delivering just that one request? What if there were no hard line between "development" and "production," but a continuum of incremental development, deployment, and support, all done by the same team? And what if management wisely treated that team not as a locally-optimized resource that had to run at 100% capacity at all times, but as a dedicated resource ready to act on a single request the moment the customer "pulled" it? The result might resemble Figure 5.
There still remains one inventory, although it has not been shown in the figures. It's the IT department's project portfolio. How can we eliminate that one? Turn it into a prioritized list of software features that align with business objectives. The main advantage of this approach is the elimination of needless work of the kind that tends to find its way into projects, since each individual feature has full visibility to management and each can be prioritized according to immediate business needs. The project portfolio allows work to remain in the inventory after its business value has declined; by the time a project team has been assembled and a project has been initiated, the requirements defined for that project may no longer be relevant to the business. It is also very easy for redundant features to hide in the details of multiple projects. The proposed alternative does not maintain a list of projects, but of specific value-add software features with no duplication. It is not a portfolio; it is a value stream.
The value stream approach enables the whole enterprise to pull individual value-add features and have them built on-demand by a single-piece flow process. It eliminates all the work-in-process inventories.
