This article reiterates and then continues the discussion of stolen time started in this post on my agile development blog.
Is the time spent on technical tasks "stolen time?"
Any software development project involves both the creation of features the user has requested and the creation of technical frameworks and supporting code that enable the solution to run. Jon Kern uses the term feature to describe some bit of user-requested functionality, and the term task to describe some bit of necessary technical functionality. Many others use the term user story for the former and the term technical story for the latter, and understand the word task in the way that Timo Rantalaiho describes it. Whatever terminology you prefer, the fact remains that both types of work are necessary to produce a working software product.
Unfortunately, some agile teams do not treat the technical stories as stories at all. They define "story" to mean, strictly, user stories — stories that describe features. Then they must confront the fact that the technical work must be done whether it is tracked as "stories" or not. To deal with that reality, some teams treat the time spent on purely technical work as "stolen time." But that leads to another dilemma: Stolen time is not supposed to be billed to the customer. Stolen time is time that is taken away from productive, project-related work for some unavoidable reason — illness, vacation, administrative tasks, mandatory staff meetings, urgent technical support issues, or some other trivial thing unrelated to the delivery of the solution. Technical tasks germaine to the project are not in this category; they are definitely project-related.
So, some teams create a special category they call billable stolen time. This provides the team with a way to bill the customer for project-related technical tasks without admitting that technical work is part and parcel of solution delivery. The fallacy is that they believe the customer either does not want to hear about it or will not understand it. But the truth is the customer needs and wants to know about everything he/she is paying for. In the agile community, we call this transparency.
I have seen the lack of transparency lead to strained customer relations. Why, I ask while wearing the customer's shoes, should I permit you to steal time from me and then bill me for the privilege of doing so? You may only bill me for work that contributes to the delivery of the features I have requested. Everything else is somebody else's problem.
At that point, the project manager tries to tap-dance around the question. The customer no longer feels like part of the team, and begins to revert to the traditional adversarial relationship with IT professionals. Lacking trust, the customer will demand more and more formal status information, and the project will soon find itself back at the old waterfall.
Personally, I don't understand why some people don't think the technical tasks are part of delivering the solution. Let's reason through it.
Okay, okay, you made your point...but what difference does it make, really? Doesn't it all wash out in the end?
The difference lies in whether the customer is really able to see where his/her money is going. Any time billable work is dumped into a general category, whether you call it "billable stolen time" or any other name, it naturally raises suspicion about how the time is being spent. Customers may not be able to understand the details of technical tasks that are necessary to make their features run, but they do understand the simple fact that some kind of "technical stuff" must be done before any software will run. They can understand statements like "Set up the database" and "Define user groups for authentication" even if they would be unable to complete those tasks themselves. By having that work on the table in the form of stories, the customer has a way to see where the time and money are going, and to understand why relatively few features are delivered in the first couple of iterations.
Let's walk through a hypothetical project. The customer has budget for four sprints. We're going to use our time machine to go back and run the project two different ways. First, we'll hide the technical stories from the customer and call that work "billable stolen time." Then, we'll be fully transparent with the customer and count all necessary work as "stories."
Our business analysts work with the customer to elicit a list of features. The features break down into nine user stories. Developers make a gut-feel estimate of each story in terms of "story points," using the Fibonacci numbers up to 13, as suggested by Mike Cohn. The customer assigns a value to each story based on his/her own reasoning.
Meanwhile, the technical team members work out the solution architecture and identify six technical stories. They estimate the stories in the same way as the user stories, but these stories have a user-defined value of zero. The technical team members also determine the dependencies between stories. We end up with a Product Backlog that looks like this:
| Story | Points | Value |
|---|---|---|
| User Story 1 | 5 | 75 |
| User Story 2 | 8 | 60 |
| User Story 3 | 3 | 55 |
| User Story 4 | 5 | 50 |
| User Story 5 | 13 | 45 |
| User Story 6 | 5 | 35 |
| User Story 7 | 5 | 10 |
| User Story 8 | 8 | 5 |
| User Story 9 | 3 | 2 |
| Technical Story 1 | 8 | 0 |
| Technical Story 2 | 5 | 0 |
| Technical Story 3 | 5 | 0 |
| Technical Story 4 | 3 | 0 |
| Technical Story 5 | 2 | 0 |
| Technical Story 6 | 1 | 0 |
The dependencies among the stories end up looking like this:
| Story | Tech Story 1 | Tech Story 2 | Tech Story 3 | Tech Story 4 | Tech Story 5 | Tech Story 6 |
|---|---|---|---|---|---|---|
| User Story 1 | X | X | X | |||
| User Story 2 | X | X | X | |||
| User Story 3 | X | X | ||||
| User Story 4 | X | X | X | |||
| User Story 5 | X | X | X | |||
| User Story 6 | X | |||||
| User Story 7 | X | |||||
| User Story 8 | X | |||||
| User Story 9 | X | |||||
| Technical Story 1 | X |
Now let's create a sprint backlog for sprint 1. We think we can deliver 30 points of work in a sprint. Based on our CSM training, we know we should apply an adjustment factor of 0.6 to the first sprint to account for ramp-up time. So, we commit to 18 points of work in sprint 1. As a starting point, based on the dependencies we've identified, we have the following:
| Story | Points | Value | Comments |
|---|---|---|---|
| Technical Story 3 | 5 | 0 | Prerequisite to Technical Story 1 |
| Technical Story 1 | 8 | 0 | Prerequisite to User Story 1 |
| Technical Story 2 | 5 | 0 | Prerequisite to User Story 1 |
| 18 | 0 |
This isn't so good, because we're not delivering any user value. An agile principle is to deliver at least some user value in every iteration, including the first. So we shift things around a bit in order to bring a user story with a reasonable amount of customer-defined value to the fore, and we end up with this:
| Story | Points | Value | Comments |
|---|---|---|---|
| Technical Story 3 | 5 | 0 | Prerequisite to Technical Story 1 |
| Technical Story 1 | 8 | 0 | Prerequisite to User Story 1 |
| Technical Story 5 | 2 | 0 | Prerequisite to User Story 3 |
| User Story 3 | 3 | 55 | |
| 18 | 55 |
Now let's run the project both ways and see what happens.
Sprint 1 when technical work is hidden from the customer
The sprint backlog that we show the customer for sprint 1 includes no technical stories, because we want to show the customer only the user stories. We'll slip the technical work in "on the side" and call it "billable stolen time." This way, the customer will think the entire universe consists solely of the features he/she has requested. They don't understand the rest of it, anyway. So we show them this:
| Story | Points | Value | Comments |
|---|---|---|---|
| User Story 3 | 3 | 55 | |
| 3 | 55 |
The customer is perplexed. He/she understands there's a certain amount of ramp-up work to do at the beginning of development. But 3 story points? Is that really all we can deliver? What are we doing with the money and time? We assure him/her that we have some technical stuff to take care of, but that real value will start rolling in very soon. The customer is skeptical, but willing to proceed. Already the customer relationship is not as positive as it could be, thanks to the lack of transparency...and nothing has happened yet!
What happens...
Due to the kinds of problems that often come up when a team is creating the technical infrastructure for a solution, the team doesn't have time to get started on User Story 3 after all. They manage to complete all three technical stories, and they have good reasons for the additional time they spent, but none of that is visible to the customer. Instead, the customer only sees an entire sprint devoted to "billable stolen time," and no value delivered. Adding insult to injury, the customer is being billed for the equivalent of 24 points' worth of "billable stolen time." The velocity chart for sprint 1 looks like this:
...and the earned value chart at the end of sprint 1 looks like this:
At the end of sprint 1, no value has been delivered, the team's velocity appears to be zero, and the customer is left feeling uncertain about whether the agile approach is going to be worthwhile. At least with the traditional methodology, there was always someone to blame. This is starting to look like a money pit.
Sprint 1 when technical work is visible to the customer
We show the sprint backlog we came up with to the customer and explain that the technical stories are necessary in order to make the requested features work. Although the customer lacks the technical experience to understand the details, he/she does understand the general reasoning behind this, and feels very positive about going forward with the project.
What happens...
The team encounters the same technical challenges as in the first case, and fails to deliver User Story 3. The difference is that the customer already knows about those problems and is aware that the technical stories took a little more work to complete than was originally estimated. Although no user stories were delivered in the first sprint, the customer has full visibility into the progress the team is making and knows value will soon be delivered. The velocity chart for sprint 1 looks like this:
Earned value is zero for the same reasons as in the non-transparent example. But this time the customer is part of the team and understands what is going on. The customer relationship is in good shape.
Sprint 2 when technical work is hidden from the customer
The sprint backlog for sprint 2 begins with the leftover work from sprint 1. Some people call this hangover. We still think we can deliver 30 points of work in a normal sprint. We attribute the low velocity in sprint 1 to specific technical problems that will not recur. Working from the product backlog, we commit to 30 points of work in sprint 2, and we assure the customer that we have solved the initial technical problems and are now poised to deliver high value. A subtle problem occurs as a result of the poor customer relationship and the lack of transparency: The customer does not really examine the proposed sprint backlog to see if there are opportunities to improve the value returned, because he/she doesn't feel fully involved in the project.
| Story | Points | Value | Comments |
|---|---|---|---|
| User Story 3 | 3 | 55 | Left over from sprint 1 |
| User Story 1 | 5 | 75 | |
| User Story 2 | 8 | 60 | |
| User Story 4 | 5 | 50 | |
| User Story 6 | 5 | 35 | |
| 24 | 275 |
What happens...
The team completes 22 story points during sprint 2, but only 8 of these count toward user stories. Six are used in the incomplete User Story 2, which cannot count at all according to the agile principle that only completed stories count. A further 8 story points are absorbed by technical work that is not being tracked as "stories." User Story 2 is left unfinished and User Story 4 and 6 are not started. The customer is billed for 14 points' worth of so-called "billable stolen time." The whole "billable stolen time" thing is getting old. Where's the promised value? Since the technical work is hidden as "stolen time," the implied value promise for sprint 2 is very high, yet very little is actually achieved. The velocity chart now looks like this:
...and the earned value chart at the end of sprint 2 looks like this:
The customer is not pleased about having to pay for "stolen time," and still does not quite see where all his/her time and money are going, but at least the team is showing a non-zero velocity now, and at least some value has been delivered, even if less than promised. The customer decides not to cancel the project just now, but this is still far from a collaborative customer relationship and the reason is the lack of transparency. Things really aren't going badly, but the lack of transparency leaves the customer with a bad taste.
Sprint 2 when technical work is visible to the customer
Although the team failed to deliver any features in sprint 1, the customer is fully aware of all the technical problems the team encountered and knows there were good reasons why the initial technical stories took longer than originally expected. Working closely with the team, the customer decides to reprioritize the stories in sprint 2 to maximize the value returned. Since User Story 3 was not started, it is safe to move it. The customer moves User Story 3 behind User Story 2, which is worth 5 more value points. The sprint 2 backlog now looks like this:
| Story | Points | Value | Comments |
|---|---|---|---|
| Technical Story 2 | 5 | 0 | Prerequisite to User Story 1 |
| User Story 1 | 5 | 75 | |
| Technical Story 4 | 3 | 0 | Prerequisite to User Story 2 |
| User Story 2 | 8 | 60 | |
| User Story 3 | 3 | 55 | Left over from sprint 1; reprioritized |
| User Story 4 | 5 | 50 | |
| 29 | 240 |
What happens...
The team completes 22 story points during sprint 2, leaving User Story 3 and User Story 4 unfinished. All the work is visible to the customer, who is not worried about the fact two stories were left unfinished. To the contrary, he/she is pleased that User Story 2 was completed, since it provides more value than the stories that were not completed. The velocity chart for the transparent version of the project now looks like this:
...and the earned value chart at the end of sprint 2 looks like this:
The customer is now slightly ahead in terms of earned value, and the team is working smoothly enough with the customer that delays and technical problems are likely to be understood and forgiven. The customer is happy with the value earned so far and is looking foward to sprint 3.
Sprint 3 when technical work is hidden from the customer
Everyone is frustrated with the project, thanks to the atmosphere of distrust that has developed as a result of the lack of transparency regarding technical stories. The team commits to 23 points of user stories.
| Story | Points | Value | Comments |
|---|---|---|---|
| User Story 2 | 8 | 60 | |
| User Story 4 | 5 | 50 | |
| User Story 5 | 5 | 45 | |
| User Story 6 | 5 | 35 | |
| 23 | 190 |
What happens...
Midway through the sprint, the customer decides to "take control" of the project. The team has repeatedly promised high value and has consistently failed to deliver. At the same time, there always seems to be a substantial amount of mysterious "billable stolen time." Enough. The customer halts work and calls a massive status meeting. For hours, everyone vents their pent-up frustration. There are recriminations and finger-pointing. Afterward, the customer insists that each team member prepare a detailed, written report of everything they have done since the start of the project. As a result, the team is only able to deliver 7 points' worth of user stories, completing the remainder of User Story 2 and all of User Story 4. They also bill the customer for 11 points' worth of "billable stolen time" for technical work. The velocity chart now looks like this:
...and the earned value chart at the end of sprint 3 looks like this:
Even after making a great effort to gain control of the project and to understand what is really going on, the customer is unhappy to be billed yet again for "stolen time." and once again to receive far less than the promised value. He/she is now considering canceling the project.
Sprint 3 when technical work is visible to the customer
The team has been making good progress and the customer is pleasd with the results as well as the process.
| Story | Points | Value | Comments |
|---|---|---|---|
| User Story 3 | 1 | 55 | Partially complete |
| User Story 4 | 5 | 50 | |
| User Story 5 | 13 | 45 | Prerequisite to User Story 2 |
| Technical Story 6 | 1 | 0 | |
| User Story 6 | 5 | 35 | Left over from sprint 1; reprioritized |
| 25 | 185 |
What happens...
The team delivers 25 story points, meeting its commitment for the sprint. The velocity chart now looks like this:
...and the earned value chart at the end of sprint 3 looks like this:
The customer and the team are enthusiastic and motivated. Everyone is looking forward to wrapping up the project on schedule and on budget with all features delivered.
Sprint 4 when technical work is hidden from the customer
This is supposed to be the final sprint, so the team accepts all the remaining user stories and hopes for the best. The customer is just hoping to realize some of the promised value for this investment.
| Story | Points | Value | Comments |
|---|---|---|---|
| User Story 5 | 5 | 45 | |
| User Story 6 | 5 | 35 | |
| User Story 7 | 5 | 10 | |
| User Story 8 | 8 | 5 | |
| User Story 9 | 3 | 2 | |
| 26 | 97 |
What happens...
The team delivers User Stories 5, 6, and 7, but is unable to complete User Stories 8 and 9. A point's worth of technical work was also done, and was billed to the customer as "billable stolen time." This is not much money, but the fact there is still work going on that seems to be project-related and yet is not properly tracked is enough to make the customer feel dissatisfied. The velocity chart now looks like this:
...and the earned value chart at the end of sprint 4 looks like this:
The customer decides to end the project at this point. Although a couple of user stories still remain, they are of relatively low value and the customer would rather live without the features than to continue working with the team under the circumstances.
Sprint 4 when technical work is visible to the customer
The team has been making good progress and the customer is pleasd with the results as well as the process.
| Story | Points | Value | Comments |
|---|---|---|---|
| User Story 7 | 5 | 10 | |
| User Story 8 | 8 | 5 | |
| User Story 9 | 3 | 2 | |
| 25 | 185 |
What happens...
The team has plenty of time to complete the remaining user stories, to do additional testing, and to prepare for production deployment. The project is completed on time, on budget, and with all features delivered. The velocity chart looks like this:
...and the earned value chart at the end of sprint 4 looks like this:
The customer is very pleased with the result and is ready to use the agile approach on his/her next project.
Conclusion
Although the results of the two projects are not radically different, the customer's perception of the process and the team's morale and sense of accomplishment are very different indeed. In the first case, the team feels as if they have "escaped" from an unpleasant project. They are relieved it is behind them, but they feel little sense of accomplishment or pride. The customer will talk about his/her experience with peers over a game of golf or a few drinks, and they in turn will be hesitant to approve a project using the agile approach for their own business units. This is a hidden cost of the lack of transparency.
In the second case, the team does not feel exhausted or frustrated at the end of the project. They are motivated and ready to start another project. The customer is pleased and excited about this high-value process called "agile development," and will talk about his/her experiences with peers over a game of golf or a few drinks. They, in turn, will be eager to approve their next projects using the agile approach. This is a secondary benefit of transparency.
My advice: Any and all work that contributes to the delivery of features must be visible to the customer, explained, and tracked transparently. Any time you call "stolen" must really be stolen — that is, it must be time that was planned for productive work but that was used up for some other purpose. On no account is this billable to the customer.