Six Sigma and Agile Software Development

A common admonition in Agile software development circles is to question everything. That doesn't mean to toss out arbitrary, unproductive questions just for the sake of "questioning." It means to assess everything we do in the context of our goals. Any activity or deliverable that doesn't contribute to the achievement of our goals is a candidate to be discarded from our process.

Now we are being asked to incorporate Six Sigma quality control practices into our software development process. The Agile principle to question everything requires us to ask pertinent questions. The main questions facing the software development organization, then, are:

1. Does Six Sigma apply to software development?

2. If so, then how?

To answer these questions, we have to understand the purpose of Six Sigma and the nature of software development. With that information, we can determine whether and how the two might complement one another.

What is the purpose of Six Sigma?

The Wikipedia entry about Six Sigma states:

This part of the definition of Six Sigma raises our first two questions:

Question 1: Is "control of variation" desired in software development?

The idea that variation is bad comes from the manufacturing industry, and more specifically from the discipline of quality control in assembly line operations. Once a widget has been designed and the production line has been set up, every unit must be as nearly identical to every other unit as possible. Otherwise, customers will not be able to count on parts fitting together properly, or different units of the same widget behaving in a predictable and consistent way.

We can conclude that if software is built in the same way as widgets, then a quality control method that works well for widget manufacturing will probably work well for software development, too. Many people have believed this to be the case, and that it is practical and desirable to create software factories to produce software solutions as if they were widgets. To illustrate the popularity of the idea, a Google search on December 5, 2005, yielded about 27,400,000 hits for the phrase, "software factory."

Unfortunately, the whole idea is wrong. One of the most egregious myths in the software industry is that the process of software development is analogous to an assembly line operation. An assembly line produces many units of the same product, and each unit is manufactured from scratch according to the same design. In that kind of operation, it is critical to minimize variation between individual units. Software is not produced on an assembly line basis. A single "unit" of the product is produced. If millions of copies are needed, then copies are made from the original. Each "unit" is not separately manufactured from scratch. Furthermore, the design is not finalized before "production" begins. Even in software development methodologies that purport to "nail down" the design up front, there are always many changes in the design as new information in learned in the course of development.

In Agile Project Management: Creating Innovative Products, Jim Highsmith makes a cogent observation:

Because of these key differences, a quality control process aimed at ensuring "repeatable" results with "variation" kept within specified limits is simply meaningless in the context of software development. Many software development projects fail, but is the answer to enforce quality control methods that simply do not apply to software development? It seems to me this is the solution to a different problem.

Answer 1: "Control of variation" is not a meaningful goal for software development, in the same sense as it is for manufacturing.

The Wikipedia entry about Six Sigma raises a second question, as well.

Question 2: How does Six Sigma apply to service operations?

The success of statistical methods in quality control for manufacturing processes has naturally led to interest in whether such methods can improve the performance of other types of business processes. Proponents of Six Sigma tend to let their enthusiasm blur their vision a bit when talking about the applicability of Six Sigma to business processes other than manufacturing operations. For example, in chapter 2 of Six Sigma for Green Belts and Champions: Foundations, DMAIC, Tools, Cases, and Certification , authors Howard S. Gitlow and David M. Levine thought it necessary to defend the use of Six Sigma for non-manufacturing processes:

The fact a manufacturing firm has a human resources department or a cafeteria is really not relevant to the quality of the goods coming off its production lines. To say many people employed at the firm are not directly engaged in the manufacturing process does not imply Six Sigma methods have a role to play in the nonmanufacturing activities of the company. There is simply no logical connection between the two statements.

If Six Sigma is mainly useful when "control of variation" is desired, then it makes sense to apply it to business processes that have that characteristic. Does it make sense to apply the method to processes that don't have that characteristic? Will it really work equally well?

An instructive example may be the Indian services company, Wipro . Wipro is a global services company offering a broad range of professional services, including consulting and software development. A year or more ago they were interested in improving the efficiency of a back-office operation handling human resources changes for a US client. They turned to Toyota for help with adapting that company's quality control method, a variation of Kaizen .

Despite initial resistance from employees, the Toyota method proved very effective in the human resources back office operation at Wipro. According to a Business Week article from July, 2005, the operation enjoyed a 43% improvement in efficiency, as well as improved employee morale. Inspired by this initial success, Wipro decided to extend the Toyota method to its software development area. But they did not enjoy the same level of success in that area. What is different? The back office operation is structurally and functionally similar to an assembly line operation. Each employee handles the same step in the process repeatedly throughout the day, handing the paperwork along to the next person in line. They sit at long tables rather than in separate cubicles. The work of software development does not follow this pattern. It requires constant critical thinking and creativity to address a series of unique problems and challenges. Requirements, desires, and priorities change constantly, in contrast to the predictable nature of the human resources paperwork proceess. The approach to quality control that worked so well in the human resources operation simply did not fit the realities of the software development area.

Granted, Kaizen/Toyota and Six Sigma are not identical. Kaizen aims at continuous, incremental improvement, while (according to Gitlow and Levine), Six Sigma "demands breakthrough improvement" (page 22). Yet, both are quality management methods based on detailed data collection and analysis, and tailored for manufacturing processes. Thus, the Wipro example is apt.

Does Six Sigma also apply to service operations? It depends on whether the approach is applied to the right factors. The process can all too easily become an end in itself, and people can lose sight of whether it is actually generating any useful results. For example, consider the following hypothetical situation from the Gitlow and Levine book, intended to illustrate how Six Sigma can apply to a nonmanufacturing industry:

There are so many things wrong with this passage on so many levels that one's first reaction to it is to think it is a sketch idea for a comedy program rather than a serious example of business process improvement.

First, and probably most obvious, is that their hypothetical situation illustrates one of the problems I mentioned earlier — the possibility we might miss the forest for the trees. The measurements are aimed at minimizing the variation in the quantity of peas consumed per patron — and that's just what Six Sigma is good for: Minimizing variation. Taking a step back to look at the big picture, it's clear that the procedure described here will be useful if and only if the business objective is to induce people to eat exactly the same quantity of green peas every day. Clearly, that is not the business purpose of a restaurant.

Second, and probably just as obvious, is that a patron who has a hankerin' for a big hunk of rare beef is not going to be especially interested in the green peas. If you conducted an exit poll and asked patrons whether they had enjoyed the side dish, many of them would probably say, "What side dish?" (see sidebar: Beans)

Third, and still well within the bounds of the obvious, is the fact that restaurants don't have time to assign busboys the task of weighing the uneaten portions from customers' dirty dishes. Busboys already have plenty of real work to do.

Fourth, there is the question of false precision . This illustrates a certain characteristic of American culture: We have an implicit belief in anything that sounds "scientific." The more data an analyst presents, the more we trust the conclusion. Even in everyday matters we fall into the trap of false precision. For instance, most Americans think the normal human body temperature is precisely 98.6°. Parents rush their children to an urgent care center the moment their temperature reaches 99°. After all, the only logical reason to specify the temperature with such precision is that the precision is important, right? Wrong. The figure 98.6 is merely a conversion from Celsius to Fahrenheit of the approximate normal body temperature, 37° C. Variation throughout the day is perfectly normal.

Our cultural obsession with pseudo-scientific precision and over-analysis carries over into most everything we do. It leads to a great deal of wasted effort, time, and money, as well as confusion about why things don't work very well despite our best attempts to "control" everything.

The fact is the chef doesn't need to know the exact weight of the uneaten peas to understand whether they are being eaten. He doesn't need to use any of his precious time to calculate the average and range of uneaten-pea-weights. He can simply glance at the returning dishes occasionally, without slowing down his regular work (in fact, this is part of his regular work), and make a mental note of which foods are not being consumed. That's all the information he needs to make a decision about menu changes or recipe changes.

Now, I hope all this has been a little bit amusing, but I also hope the real message isn't lost in the frivolity. One of the main problems with highly structured and formalized methods of quality control is that people often measure the wrong things or measure things in ways that don't help them achieve their true goals. They end up measuring just for the sake of measuring, and reporting just for the sake of reporting. The process becomes useless overhead that never yields any benefit. Worst case, people convince themselves and others to take entirely the wrong course of action.

The cited authors both hold PhDs and both teach business administration and statistics at the post-graduate level. They are not fools. But formal education doesn't protect them from their own human nature, and like many of us, they have a tendency to overlook the pitfalls in things they admire. They have come to value the Six Sigma method because they have seen it bring positive results in so many cases. For the best of reasons, they want to apply the method to even more situations. But what works in one circumstance doesn't necessary work in another. I think it is instructive to note that the best example these experts could come up with to demonstrate the value of Six Sigma in nonmanufacturing processes actually shows the opposite. The restaurant could spend a lot of time and money analyzing the wrong factors in excruciating detail, only to reach incorrect conclusions and lose money.

Answer 2: Six Sigma applies to any sort of business process that has characteristics similar to those of a manufacturing operation, including some service operations. It does not apply as well to other sorts of business processes.

Is software development a manufacturing process?

It seems clear that if software development is a manufacturing process and a business process, then we may be able to apply Six Sigma to improve quality. Logically, if software development is not a manufacturing process or not a business process, then Six Sigma may not be a good fit.

Many people have lived and died by the belief that software development is just another form of manufacturing. Just look at all the attempts to realize the idea of a "software factory."

I keep saying that software development is fundamentally unlike a manufacturing process. If you are questioning everything (as you should be), then you must question that assertion, as well. If software development is unlike manufacturing, then what is it like?

Question 3: What sort of process is software development?

Although mainstream thinking in IT management still does not seem to accept the idea, many technical professionals have seen parallels with construction for many years. The most obvious evidence of this is the use of design patterns in professional software development. Software developers see design patterns as powerful and elegant tools for building solutions, because they fit so naturally with the realities of software development. The whole idea came directly from architect Christopher Alexander's book, The Timeless Way of Building , which describes the results of his work in the 1960s and 1970s in urban planning and architecture. He writes,

That quote captures the essense of the difference between manufacturing and software development. The similarities between software solutions are at the level of the general patterns that may be observed in the "environment." The individual solutions themselves are quite unique. In software development we use a solution "a million times over, without ever doing it the same way twice." A quality control mechanism that seeks to minimize variation applies to a process in which the solutions are done the same way twice, ten times, or a million times. Therefore, such a mechanism is fundamentally at odds with the basic nature of software development.

The Standish Group , specializing in risk and cost analysis for the IT industry, publishes a report about the success of software development projects called the Chaos Report. They categorize IT projects into three classes based on how well they meet their requirements, stay on schedule, and stay within budget:

• Resolution Type 1: Success. Completed on time and on budget with all requirements met.

• Resolution Type 2: Challenged. Completed and operational, but went over budget and/or time and does not support all the features originally required.

• Resolution Type 3: Impaired. Cancelled prior to completion.

According to the original Chaos Report , published in 1994, only 16.2% of IT projects in the US were successful, while 52.7% were challenged and 31.1% were impaired/cancelled. Personally, I would consider the challenged and impaired projects all to be failures. Thus, as of 1994, some 83.8% of all IT projects failed. The Chaos Report for 3Q 2004 finds 53% challenged, 18% cancelled, and 29% successful. By my measure, that's a 71% failure rate — slightly better than in 1994, but still dismal.

In the introduction to the first Chaos Report in 1994, the authors raise the analogy between the construction industry and software development. They cite a 1986 article in which Alfred Spector, president of Transarc Corporation, compared software development to bridge-building:

Construction analogies seem to come naturally to people's minds when they try to describe the nature of software development. A software development practitioner can see similarities even when the author/speaker knows nothing about software development and was interested in an entirely different subject. That was the case with design patterns, and also with the article by Alex Laufer published on PM Forum for May/June 2005. The article, "Learning from Experience," summarizes Dr. Laufer's experiences when he undertook an analysis of construction projects for the Construction Industry Institute (CII) Project Organization task force . They enlisted his help to try and understand why some projects succeed and others fail. His findings resonate with software development, although he was not addressing that topic and apparently has no background in IT. He writes,

The parallels between construction and software development are clear. We experience a high failure rate because we have worked under the assumption that it was possible to achieve a high degree of certainty about what we were building and how to build it before development was even started. This is the "predictive" approach to software development in a nutshell — predict the risks, the costs, the timeline, the requirements, the software tools, the methodology all in advance and in detail. It doesn't work because it's based on a myth — that it's even possible to know those things with certainty in advance. It's the myth that has led our industry to its present 71% failure rate.

A manufacturing operation has very different characteristics. The design of the product is finalized before the production line is set up for the simple reason that the widgets are real objects. It is not possible to "discover" any element of the design — however minor — after production is underway, as is normal with software development. Everything about the widgets must be known in advance; with software — especially business application software — it isn't even possible to know everything about the solution in advance.

Answer 3: Software development has more in common with construction than with manufacturing.

Is software development a business process?

The purpose of Six Sigma is to gain control of quality in business processes. Often, when Six Sigma or other statistical quality control methods are introduced at a company, people try to apply the new method to every activity in the organization. But the purpose of these methods is to assure the quality of business processes. When they are applied to activities that aren't really business processes in their own right, it usually leads to useless administrative overhead with no business benefit.

As Six Sigma becomes insitutionalized at our company, we must determine whether it should be applied to our software development process. One way to approach that question is to ask whether software development is a "business process" in its own right. If it is a business process, then it should be managed in the same way as any other business process. If it is not a business process, then we must avoid imposing useless administrative overhead on it.

Most business operations these days include some level of software support. Practically any interaction you might have with a company involves the use of computer software. A customer service representative may access a business application to answer your query. A telemarketer's pitch is supported by a business application. A bank teller uses a business application to handle your transaction. When you buy gasoline at the pump with a credit card, a business application performs the credit authorization. Many back office operations consist of automated workflow processes that involve both business applications and human steps. Software plays an integral part in all these business processes and many more.

Does that mean the business application software itself is a "business process?" Clearly, the answer is no. The applications are just tools used in the business process, like a telephone. They are not a "business process" in their own right any more than your telephone is a "business process."

What about the process by which the business application software is developed? Is that a "business process?" At companies where statistical process control methods are being implemented, there is a tendency to try and apply the methods to everything the company does, including operations that are not part of the core business activity of the organization. How does the process of software development fit into the picture? Is it a business process? Should it be subject to the same controls as other business processes?

I address this question in a little more detail below. In brief, I think the features of a business application software solution are important, because the features support a company's business operations. However, the methodology by which the software package was originally built has no bearing on the quality of the business operations the product ultimately supports in the field. The software development activity itself is not a business process from the perspective of the end user.

Is Six Sigma "scientific" (and does it matter)?

[In our society] we believe in science — and the irony in the choice of words is intentional.

According to the instructional material prepared for our company by our Six Sigma helper, Valeocon Management Consulting , the basic cause-and-effect formula is:

Y = f(X₁,X₂,X₃,X₄,...X_n)

where Y is an effect (or a desired outcome) and X are its causes. This simple formulation doesn't preclude an arbitrarily complex function f, that could even go as far as analyzing emergent behaviors in a chaotic system. But just imagine the time and effort required to analyze business processes in that way! There are two obvious pitfalls in this approach: (1) It would be easy to miss the forest for the trees, and (2) by the time you had completed your analysis, the business opportunity for which the project had been proposed may have passed.

We live in a society that values a scientific approach to problem solving. More to the point, we believe in science — and the irony in the choice of words is intentional. Few Americans have much understanding of scientific method or formal logic, and fewer still actually apply those skills in their everyday lives. Certainly, business processes are not based on comprehensive analysis or cold reasoning. More often, they originate by trial and error and become established through simple force of habit. Yet, we like to think of ourselves as a scientific people. When someone comes along and offers a solution that looks and smells scientific, we tend to accept it. Based on painstaking data collection, elaborate mathematical formulas, and arcane statistical methods, Six Sigma certainly has the look and smell. But is it really scientific and do we really care?

Question 4: Is Six Sigma scientifically rigorous?

Gitlow and Levine assure us that "Six Sigma projects are far more structured and formatted than projects in most previous quality management processes." (I can't help thinking that highly structured and formatted project management approaches have led to the 71% IT project failure rate our industry currently enjoys, and to be reminded of Albert Einstein's definition of insanity: To repeat the same action and expect a different result. But that's neither here nor there.)

Six Sigma theory is based on statistical analysis techniques. Statistical analysis has proven valuable in many industries, from gambling to manufacturing. The Wikipedia entry about Six Sigma offers the following explanation of the scientific statistical basis of the method:

So, it seems Six Sigma is not so much a mathematically rigorous method for assuring quality as it is a cultural mindset focused on quality improvement. To that extent, at least, there is no inherent conflict between Agile software development and Six Sigma management. But the devil is in the details, as the saying goes.

Answer 4: Six Sigma is more a matter of mindset and approach than of pure scientific rigor.

Sounds like the English word six was chosen to alliterate with the word sigma and because the Arabic numeral 6 looks pretty next to the Greek letter σ. The "science" basically goes downhill from there. But does that matter, provided the method yields the results we want? Let's think it through.

Question 5: Do we need scientific rigor to reduce the IT project failure rate?

Let's start by doing what IBMers used to call a "level set;" that is, let's reiterate our goal. What we really want to do is reduce the appalling failure rate of IT projects and arrive at a way of doing our work that assures customers they will receive high quality results that meet their requirements on time and on budget.

The problem with that approach is it takes too long. By the time you have an answer, it's too late to put the answer to use. In a competitive business environment, that is simply untenable.

Here is a small example of what I mean. A couple of years ago at our company, a lending unit wanted to provide field representatives with a wireless client they could use to enter and query loan applications. They felt it would help them approve loans faster, and increase revenue. The trick was that the peak sales season for the industry segment was coming up in a couple of months. If we missed that time, the business unit manager believed we would lose marketshare. There was significant opportunity cost involved in any delay. We chose to use an Agile approach to the project rather than a traditional approach, because the traditional development group estimated ten months to complete the work (we did it in 6 weeks with a team of 4 people). In addition, the traditional group wanted to build a certain piece of technical infrastructure to make the solution cleaner, and that would have increased the cost by a significant amount. At that point, the business unit manager thought the solution would yield increased revenue, but was not sure of that, and she did not want to spend much money on the solution. There was no time for a rigorous, paperwork-intensive development process, because by the time such a process was complete it would have been too late for the solution to be useful, no matter how well-designed and implemented it might be. It turned out that the solution has not really increased revenue by all that much; therefore, we did the right thing by keeping development costs to a minimum.

There are many problems in the world that have similar characteristics, especially in areas of public policy pertaining to environmental stewardship and sustainable development. If a factory is dumping pollutants into a wetlands that, in turn, filters water that is ultimately used as a human drinking water supply, we simply lack the luxury of time to allow for a comprehensive and thorough scientific analyis of all the factors involved before a public policy decision must be made. This sort of situation has given rise to the notion of post-normal science . The Wikipedia entry states:

This is often the case with software development projects to support business processes. Indeed, it is one of the main issues that prompted the Agile movement. Facts are uncertain, values are in dispute, stakes are high, decisions are urgent, and there is simply no time to follow a time-consuming, rigorous, "heavyweight" process.

And it is a point of conflict between Six Sigma and Agile software development practices. Six Sigma is based on the kind of thinking described by Thomas Kuhn as "normal science," in his influential book, The Structure of Scientific Revolutions . It's part of the culture of believing in science.

Answer 5: We need scientific rigor for certain problems, and a post-normal approach for others. One approach must not trump the other simply by definition.

How do we reconcile Six Sigma with Agile software development?

There is often a cultural conflict between adaptive/Agile software groups and process improvement groups in organizations that are trying to implement a method such as Six Sigma. Objectively, however, there is no fundamental conflict between the two. Agile development focuses strictly on satisfying customer-defined requirements with a minimum of administrative overhead. Six Sigma focuses on satisfying customer-defined requirements by applying rigorous process controls. The difference lies in the implementation; the end goal is the same. Therefore, there may be little or nothing to "reconcile." It may only be a question of understanding when to apply each approach, and at what level to incorporate each into the standard practices of the organization.

Compare the way an Agile software development project typically runs with the way Six Sigma approaches the problem of improving an existing business process. The Six Sigma DMAIC process closely parallels the Agile project approach. The Define in DMAIC means to quantify the desired results based on the voice of the customer, a critical concept in both approaches. The Measure in DMAIC means to keep track of which causes are contributing to which effects. The Improve in DMAIC means just what it says. The Control in DMAIC is analogous to the Agile concept of continuous learning, well known in organizational science as double-loop organizational learning . There is nothing unfamiliar or contradictory here, at least at a conceptual level.

Six Sigma and Agile development are both dedicated to reducing failure rates and improving customer satisfaction. They represent very different mindsets about how that is to be accomplished, though. In answer to the failure of heavyweight process controls to assure IT project success, Six Sigma imposes even heavier process controls in order to gain a thorough understanding of cause and effect. The Agile approach is based on an entirely different philosophy; reducing formal controls in order to give qualified professionals the freedom to apply critical thinking skills and creativity to unique problems. The two approaches are effective in different situations.

In a limited domain such as a manufacturing process, it makes perfect sense to gain a thorough understanding of cause and effect. In a service operation it may make sense, too, depending on whether it is important to minimize variation in some factor (remember that the main purpose of Six Sigma is to minimize variation). An example often used in Six Sigma training is a pizza delivery service that wants to keep delivery times within the bounds promised to customers and ensure the pizzas are all consistent — height of shell, evenness of diameter, color of shell. It is possible to analyze all the factors that contribute to these quality factors and identify areas for improvement.

You can see that the restaurant example cited earlier in this article represents a very different problem in a very different business context, although both examples have something to do with food service. It is important to understand the characteristics of the business operation to be measured in order to choose an appropriate method. It is easy to assume that any approach that worked in one food service situation must work in another; or that any method that worked in one information technology situation must work in another. That isn't necessarily true.

Software implementation and software development are not the same type of problem. Six Sigma is an effective method for software implementation projects in particular.

Statistical process control methods have a long history, going back to the work of W. Edwards Deming some 50 years ago. Of the many SPC methodologies , the best known and most widely applied have been Total Quality Management , Kaizen , and Six Sigma . When applied intelligently to the types of problems they are meant to address, SPC methods can yield significant benefit.

A website dedicated to the application of Six Sigma to information technology includes an article by Gary Gack, a founding partner of Six Sigma Advantage , summarizing the Six Sigma approach to software implementation projects. The article shows there are many fundamental similarities in the philosophies of Agile and Six Sigma, as applied to software implementation.

An interesting point, and one we should not overlook, is that the author distinguishes between software development and software implementation projects. The author claims Six Sigma is an effective method for software implementation projects in particular; not for software development projects. This, from a man with 40 years IT experience, a Six Sigma Black Belt, an ASQ-certified software quality engineer, and an MBA from the Wharton School.

Question 6: Should we manage the software development process as such with Six Sigma?

In an earlier article, I define three different relationships that companies have with information technology: A primary technology company creates new hardware and/or software, a tertiary technology company uses those products to support its own business operations, and a secondary technology company helps tertiary companies figure out the products offered by primary companies. The process of software development is a business operation in its own right only for primary technology companies; for all others, it is simply a support function. Any method that treats software development at the same level as a true business operation is going to miss the target to some degree.

Based on the preceding discussion, any process that is both a business process and a manufacturing process may benefit from the Six Sigma method of quality control. This is shown by the following truth table :

BP	MP	BP AND MP
F	F	F
F	T	F
T	F	F
T	T	T

We have shown that software development is not a manufacturing process in any case. We have also shown that software development is a business process only when the core business of a company is software development. For secondary and tertiary technology companies, software development is only a supporting activity, and not a business process in its own right. Finally, we have shown that the underlying philosophies of Agile development and Six Sigma management are different. The former seeks to minimize administrative overhead to leave professionals free to do the right thing, while the latter depends on formal documents, hand-offs, and abstract data analysis to enforce quality on a process from without. Therefore, the question of whether Six Sigma applies directly to an Agile software development process in a tertiary technology company is answered by the first row in the truth table.

But we still don't have a final answer. The consultancy that is helping us implement Six Sigma has a simple decision flowchart to help with the selection of Six Sigma projects. It looks something like this:

Figure 1: Choosing a Six Sigma project

The flowchart appears straightforward, but in reality it is subject to a great deal of interpretation. For example, the assessment of a project idea is different if the idea pertains to business process improvement than if the idea pertains to a software development project. Unless the company is a primary technology company, the software development process is not a business process in its own right; it is only a supporting function. (The features of a software package are important to the business process, but the way the software was built is irrelevant.) Thus, the level of detail of the two types of project ideas is quite different.

Let's modify the flowchart a bit to make it clear that we are considering ideas for business process improvement as candidates for Six Sigma, rather than just any and all project ideas:

Figure 2: Does Six Sigma fit a business process improvement idea?

In addition, the understanding of terms varies depending on the background and area of focus of the person reading the flowchart. A business person assessing a business process will have a different understanding of "Solution Known" than a software development professional assessing a software project idea. To the former, "Solution Known" means that the entire end game is fully understood, and therefore there is no need to apply Six Sigma or a similar method to learn how to improve the business process. To the latter, "Solution Known" may only mean that the approach to discovering a solution is known. That is simply part and parcel of professional software work, and does not represent an "unknown" that requires statistical analysis. This is especially true when Agile methods are used.

In an organization that uses both predictive and adaptive methods for software development depending on the characteristics of a given project, the assessment is still more variable. In that case, "Solution Known" may mean even less than in the simpler cases. It may mean nothing more than we "know" we have to apply another decision-making approach to the problem. For example, if we are working in a tertiary technology company and we are assessing a software development proposal, as opposed to a business process improvement proposal, then "Solution Known" means we can assess uncertainty and urgency to choose either a predictive or an adaptive approach to software development. To a software professional, this is a case of "Just Do It." There are further decisions to make regarding the choice of approach, choice of methodology, and so forth, but to a software professional this is all "known" activity.

To a business professional, it does not seem quite so secure. Instead, it sounds as if a great deal is still not known. If we are assessing a proposal for business process improvement rather than for software development, then the business professional would be correct. The approach to discovering the right solution to a business process improvement problem is to apply Six Sigma or a similar method. It is an entirely different type of question than in the software development case. It is very important, then, to understand clearly and objectively whether you are assessing a business process improvement project idea or a software development project idea.

Obviously, since nearly all business processes are supported by application software, software development projects will fall out of any business process improvement initiative. That is only to be expected. However, each of those development projects is not a business process in its own right, and need not be managed in the same way as the overarching business process is managed. That would be a misapplication of Six Sigma.

Another decision point is whether the problem is the result of a repetitive process. We have already explained that software development is not a repetitive process in the same sense as a manufacturing process. It is more analogous to the construction industry, in which certain well-known engineering principles are applied to every project, and certain well-known design patterns may be applied, but each project is unique — with the exception of manufactured housing components, there is no assembly line process on a construction project. In software development there is absolutely no assembly line process, since any given piece of software is only developed once. When a software development project is being assessed by this flowchart, the answer to this question must always be "No." However, a business process improvement initiative may be inspired by a problem in a repetitive process, and the solution may, in turn, spawn a software development process. It is important in this case to understand the differences between business processes and software development processes. In particular, we must keep in mind that the two are really at different levels of detail.

Is it ever appropriate to apply Six Sigma to a software development process? Yes. In a primary technology company whose core business model includes software development along with marketing and/or consulting services, the software development process is a business process. The role of software development activities in a primary technology company is very different from that in a secondary or tertiary technology company.

For example, consider the software firm S1 , specializing in enterprise software for the banking industry. S1 uses a predictive (waterfall) software development methodology based on the Rational Unified Process (RUP). Their customized RUP process is transparent to customers (they can view the bug tracking system directly), is based on frequent, direct communication among all team members, and enjoys the advantages of iterative development. But it is a predictive process overall, since the product line must meet known regulatory requirements, support industry best practices, and interface with other mainstream banking software products. Since software development is the company's core business practice, and since a predictive development approach lends itself to formal process controls, and since S1 develops its product line on a fixed release schedule (thus making their development practice a repetitive process), S1 is an organization that could apply Six Sigma directly to its software development process. I don't know whether they do so; I only cite this as a counterexample to the software development process used at a tertiary technology company, to show that some software development processes may benefit from Six Sigma while others may not.

Here is a version of the decision flowchart that takes into account the fact software development is a business process for some organizations, and only a support function for others.

Figure 3: What approach should we apply to a software development idea?

Answser 6: It depends.

If software development is a core business practice and the development follows an approach based on documentation and checkpoints, then yes. If software development is a support function and/or the development approach is based on direct communication and informal methods, then no.

Conclusion: Ask the right questions or get the wrong answer

The value of the answer one receives from an oracle, whether ancient or modern , depends on the formulation of the question. "Their answers were carefully doubled - and might have one of two or more meanings. This was certainly wise, and indicated that the oracle was not a fool, whatever might have been thought or said of its questioners." It is very easy for the questioner to fool himself by asking the wrong question, or by asking the right question in the wrong way, or by being too eager to interpret the answer in the way he desires. We must be careful of this pitfall when asking whether Six Sigma applies to any given process in our enterprise.

One can argue that business application software is a critical part of any contemporary business process even in a tertiary technology company. That is certainly true, but does it imply anything about the applicability of Six Sigma to the company's internal software development process?

Many business processes rely on telephone technology as well as on business application software. A back office operation that processes human resources requests or handles paperwork for loan servicing uses telephone technology extensively. What do we need to know about the telephone system to help those operations improve quality? The telephone system must support the features the workers need to do their jobs efficiently. Do we need to know about the telephone manufacturer's assembly line process? Not at all.

The same reasoning applies to the business application software that supports those business processes. We need to know that the software supports the features the workers need to do their jobs efficiently. We do not need to know anything about the software vendor's development process. That is outside the scope of the business process we want to measure and control.

To try and measure the software development process at a tertiary technology company is tantamount to measuring the manufacturing process at the company that makes your telephone, and then believing the information will give you insight into the quality of your own business processes, which have nothing to do with making telephones. It would merely be weighing the leftover green peas.

The bottom line is that Six Sigma and Agile development methods are two different tools that apply to two different types of problems. Neither is a panacea, and neither should trump the other as an organization's sole approach to problems.