Dream Week - a tool for agile innovation

A team that I have been working with for a while now recently came up with, and executed, a great idea for inspiring innovation and improving collaboration in an agile environment.

It is worth noting that this has only been performed by the team once, so I am making some assumptions about how it may be executed in the future. The goal of this post is not only to document what happened but also provide a framework for moving the technique forward in the future.

What is the context of the product team?

Everyone works in different ways, agile or not, so here is a bit of context about this team that you can bear in mind if you want to apply the idea to your organisation.

The product team consists of two delivery / feature oriented sub-teams. Each of these teams contains programmers, a UX professional and a business analyst. There is also a product owner (from the UX discipline), a chief architect and another UX professional 'floating' across the two delivery teams. The entire product team works on two week sprint cycles, performing a demo and retrospective at the end of each sprint. A product release is performed every few sprints.

What is Dream Week?

At the beginning of a new release one week is set aside as Dream Week. Each member of the team comes up with ideas for work to be performed within the week. The only limitations for the ideas are that they must benefit the product and must be deliverable within the week. The ideas are then pitched to the product owner at the beginning of the sprint. The whole team helps to work out a common understanding of the idea, during which time the idea may change and go through a number of quick iterations. Once all the ideas have been pitched the product owner prioritises the ideas and then the team cracks on with delivering them in priority order.

The ideas that are pulled into the sprint are built to production quality and go into the product. Ideas do not have to be product features, they could be paying back technical debt or investigation into future technologies. However, the ideas are prioritised by the product owner, so just like every day stories, the payback of technical debt must be justified against improved product capability.

By running Dream Week at the beginning of a release it gives the team and the product owner the most freedom to follow potentially controversial ideas as any mistakes made can be corrected throughout the release. It is the least risky time to innovate.

Why only a week?

"Constraints breed creativity"

http://37signals.com/svn/archives2/constraints_breed_breakthrough_creativity.php

For many people given a free reign to come up with new product features, wild and fanciful ideas start to bubble up to the surface. This is a good thing. But that creativity needs to be harnessed. By requiring the idea to be implemented within a week, the would be innovator needs to pare down the idea and get very creative about the potential solution to allow the goal to be achieved within one week.

"A change is as good as a rest"

By using a shorter time period than the team is used to there is a buzz of excitement about the weeks work. The regular heartbeats of agile iterations are useful for creating a rhythm that the team can get into and create useful touch-points for the rest of the organisation. However, the change of pace and focus for a short period is good for getting the collective heart pumping.

How does it improve collaboration?

Anyone can suggest an idea, programmers, UX practitioners, business analysts, but everyone needs to be involved to deliver. The ideas are fresh so anyone has a chance to get involved to shape the solution. The time scale is short so everyone has to pull together to meet the deadline. Because the idea has originated from within the team there is an increased sense of ownership.

Dream Week creates a vacuum of time that must be filled by work generated within the team. It's more than likely there will be too much work to fit within the time. Sound familiar? In this environment your idea is competing for space with all the other ideas being presented, so you better sell it well. Figuring out the needs of the person you are selling to, and pitching the idea, is a great way to improve communication skills.

Did it work?

The team certainly seems to think so. We fixed a number of usability issues that had been hanging over the product for a while and introduced a new pattern to achieve a user goal in a much more effective manner. The teams enjoyed the process, the product owner is happy with the delivery and the next Dream Week is scheduled for the first week after the next release.

The team have actually decided to call it Innovation Week as this makes it an easier sell to business sponsors, but I'm personally a fan of Dream Week as innovation has become almost as overused as Agile.

XPDay 2009 - Fostering collaboration between developers and UX

I convened an Open Space session at XPDay 2009 on the subject of fostering collaboration between developers and UX in an agile environment. Participants in the session were invited to tell a story about their experiences of working in an environment where software developers and UX professionals worked together to deliver a product. The stories could describe a positive or negative experience, and anything in between. We then pulled out and recorded positive and negative properties from these stories. These properties are recorded here.

None of these stories were thorough experience reports, but they were, to the best of my knowledge truthful and an accurate representation from the perspective of at least one person.

Negatives

Solutioning stories too early

An emerging pattern when integrating UX and Agile development is to have the designers work as part of the customer team to help drive the prioritisation of what gets built. This was a common theme for most of the stories, and seems to work well. However one story highlighted the need to be wary of performing too much design work before passing the user story to developers to implement. The developers in this story started to feel disenfranchised with the solution and unable to make suggestions for changes to the design.

UX is a more volatile resource

Another story told of having UX practitioners working in sprint with developers, only for the UX resource to be frequently pulled out of the sprint to work on other tasks with the customer. This had the effect of obstructing the flow of work as the developers were unable to complete the story without help from the UX practitioner.

BDUF was a source of conflict

This is a more extreme example of a UX team solutioning stories too early. In this case the UX team would produce large amounts of design documentation for each feature to be implemented, and then hand this documentation over to the developers to build. This ended up being a source of conflict between the developers and the UX team as the developers did not want to read through lots of documentation to build the feature. To get round the problem business analysts would 'paraphrase' the documentation to let the developers get on with building the features, which understandably, caused a lot of friction between the various groups involved.

Positives

UX contribute to definition of 'done done'

UX practitioners on the team contributed to the definition of the 'done done' criteria for a story. This means the values and the overall integrity of the experience of the product can be controlled by UX practitioners on the team. In addition, every single person involved in delivering the product is acutely aware of the standards that each and every feature needs to be delivered too.

UX sitting with devs and working in sprint creates a good flow of work

More than one story told of successful working environments when developers and UX practitioners sat together. One story in particular told of a transition from the UX members of the team moving from sitting with the rest of UX people in the company to sitting in the same space as the developers. While this transition was difficult in the short term the team experienced a much improved flow of work as it was easier to coordinate the different tasks that needed to be performed by developers and UX for the completion of a story.

UX working in sprint improves understanding and empathy

Having UX and developers working together throughout a sprint to achieve a common goal gives a real opportunity to gain a deeper understanding, even a level empathy, into the effort that is required to deliver a story from both the perspective of developers and UX practitioners. The specific example given in one story was that, by being involved in the estimation process, UX understood better the amount of developer work required for certain solutions.

Allow developers to collaborate on design without forcing it

We were fortunate enough to have Jeff Patton with us in the session. He relayed the story of an experience report at Agile 2009 Feature Teams - Collaboratively Building Products from READY to DONE. This company moved from a process of designing and planning work separately from the development team to what sounds like a 'by the book' scrum process of performing all activities necessary to define the work, design and deliver the required features within a single sprint. This allowed the developers to be involved in user interaction design as it became just another activity to be worked through on the road to delivering a feature within the sprint. The interesting point for me in this approach is that not everyone had to be involved in the UX work. For example, developers that were happy not to get involved in the solutioning of stories were free to work on repaying technical debt while the solutions for the stories were being worked out. Thus giving the opportunity for design collaboration if the developers wanted to provide input, but without forcing the issue.

Innovation Sprint \ Dream Week \ FedEx Day

Three ideas around the same theme, giving a delivery team more ownership of a solution and improving collaboration and innovation all tumbled out at the end in quick succession. The innovation sprint is something that people at Yahoo were toying with, please correct me if I'm wrong here. Every fifth sprint is an innovation sprint where the team get to decide what to work on. The idea being to try and introduce innovative, potentially risky ideas, that would not normally be prioritised by the product owner. I'm not sure if the result of this work is committed and deployed or if the work is all prototypal in nature to feed future stories? Interestingly the amount of time this takes away from delivering customer features, is 20%, the same amount of time as the Google Innovation Time Off.

Dream week is something that the team I am currently working with has come up with. The idea being that, at the beginning of the week anyone in the team can suggest an idea for something to work on that will add to the value of the product. These ideas are pitched to the product owner who prioritises them to be worked on and delivered within the week. Keeping the time available down to a week increases the focus of the solutions and reduces the risk.

Taking the focus on innovation company wide, another company would hold FedEx days. The idea being that you had to deliver something at the end of the day. The entire company would be arbitrarily grouped into pairs for the day. Each pair had to deliver something of benefit by the end of the day, The pairs can choose their own thing to work on, or can pick ideas out of a hat. The pairings are arbitrary so you can end up with anything from product owners and developers pairing to sales people and UX practitioners pairing. Of course something of benefit may not be software.

Explore many ideas up front

Another suggestion for improving the collaboration between developers and UX practitioners was to follow an approach of exploring more options up front, together as a team, and then focusing in on one option for delivering the solution. I believe this was given as an alternative to gradually iterating towards the solution through frequent delivery.

I would like to express my thanks to everyone who participated in this session. I thoroughly enjoyed hearing all of the stories and your experiences. I was glad to hear of so many positive experiences working on products with heavy UX and development involvement.

A number of the stories come back to what I have been learning over the past couple of years, which is, there seems very little cause for irresolvable conflict between agile developers and UX practitioners and in fact, the solution to many of the problems can be found by taking a step back and reviewing the principles of the agile manifesto and tailoring the implementation of these principles to your own environment and context.

Here are a few of those principles that I see as being particularly relevant to the positive experiences that were reported during the session:

* Business people and developers must work together daily throughout the project.

* Build projects around motivated individuals. Give them the environment and support they need and trust them to get the job done.

* The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Take a moment and have a look at the rest of them.

Accolade Passes Accredit UK Standard

Back in the summer, Accolade Consulting signed up to the Accredit UK standard in the specialism of Software Product Design and Development. Since then we have been beavering away, tweaking our processes and getting the evidence together for the site inspection.

I am delighted to announce that Accolade has passed the certification process and is now fully accredited. The accreditation is recognition of the technical excellence we bring to bespoke software development projects and the sound business practices that we have put in place.

The goal of the Accredit UK standard is to give customers confidence that they are dealing with credible companies supplying ICT solutions. While this is useful for ICT in general, I believe it is invaluable for customers looking for bespoke software suppliers. Building bespoke software can be an expensive investment and customers need to be able to trust the supplier they are dealing with.

Find out more about Accredit UK at http://www.accredituk.com.

Book Review: Debug It!

The Pragmatic Bookshelf has released a new book into beta called Debug It!: Find, Repair and Prevent Bugs in Your Code, by Paul Butcher. Paul is a friend of a friend, I don't know him personally.

I like the premise behind this book. When I showed it to a friend on the train yesterday while reviewing it, he joked that we should just code it right in the first place! We all know that even on the best teams, the reality is that bugs happen. But, as Paul points out in his introduction, there aren't many books that cover debugging. Given that debugging can be a frustrating exercise, having a resource like this is a huge benefit to the professional software developer.

Paul asserts that debugging is an 'intellectual process', it's more about the approaches you take to identify and then resolve issues than the tools that you use. Despite covering a number of popular tools and techniques that make finding and dealing with bugs easier, the message is that you have to get the mindset right. He defines the core debugging process as:

• Reproduce
• Diagnose
• Fix
• Reflect
  

From these basics, Paul covers a set of approaches to help with each of these steps in the process. A lot of the approaches will be second nature to experienced developers, although it's good to be reminded now and again of some techniques that you can fall back on if a bug is proving to be particularly tricky to root out. The book is split into three sections:

• The Heart Of The Problem - all about the basics of debugging and the mental approaches required
• The Bigger Picture - tracking and managing bugs
• Debug Fu - some of the more interesting and advanced issues you have to be aware of when debugging and resolving bugs
  

One of the best features about this book is the number of real world debugging anecdotes that are told throughout the book. Apart from being amusing stories, the bizarre nature of some of these bugs should make anyone think twice before selecting the 'could not reproduce' option on their bug tracker.

In the Pragmatic Zero-Tolerance section Paul presents a great diagram that illustrates the effect of leaving bug fixing to the end of a project. He then provides some good rational for fixing bugs early. There are a number of team approaches described for 'Digging Yourself out of a Quality Hole', but no suggestions on how to manage early fixing of defects before you get into the hole. For the record, two approaches I have worked with are:

• In an XP environment, have a bug pair, that clears all the outstanding bugs in the iteration before starting on any user stories. Rotate the bug pair every iteration.
• In a Scrum environment, whenever a story is finished, the pair pick up a must fix defect before moving on to their next story.
  

There is a nice discussion of concurrency bugs and some techniques for identifying and fixing them. He agrees with other writers on the subject that simplicity is generally your best bet. Reduce the parts that are concurrent as far as you can.

A personal favourite of mine, and I was starting to worry that it wasn't going to come up, is the section entitled, Don't Be Too Quick To Point The Finger. This is aimed at developers that assume they couldn't possibly have made a mistake, it must be that 3rd party library that is broken, not their perfect code! I couldn't agree more with his sentiment that you should trust 3rd party libraries until you have proved, beyond a shadow of a doubt that it is not your code that is buggy. The library has probably been tested a lot more thoroughly than yours.

There is a real gem of a chapter in this book, which is all about assertions. When to use them, what to use them for, how not to use them. If you're after a short guide to the why and how of assertions, this is the chapter for you. Now here's the really good news -- this is available as a free extract.

There are some interesting organisational anti-patterns given in the last chapter, of which I have certainly seen a few so it is nice to see them documented from a debugging perspective. There was no specific mention of code anti-patterns, there's a section called Assertion Abuse, which strike me as anti-patterns for assertions. But a discreet set of code anti-patterns that are likely to lead to bugs would be useful. Some mention of code complexity on the effect of defects would have provided an interesting chapter.

Overall, I think this is a really useful book. It does a great job of setting the scene for debugging and getting you into the right mind set, while also talking about the complications that can arise once the bug is found and squashed. It's almost worth looking at for the anecdotes alone, to understand the lengths that you sometimes have to go to when trying to understand some truly bizarre defects.

Agile and UCD article

My latest article on integrating Agile and UCD (Agile and UCD: Building the Right Thing the Right Way) has been published on DevX. In the article Darius Kumana, my co-conspirator, and I talk about the benefits of combining the two approaches, such as: using the frequent delivery of agile projects to help gather user feedback, and improved collaboration through collective responsibility.

We also sound a warning about two key issues inherent when integrating agile and UCD: loss of context and planning to iterate so teams can react to new findings from user testing.

Have a read and let me know what you think.

Look out for our next article on DevX where we illustrate the importance of retaining the context of use on an agile project and discuss techniques from UCD to help communicate the users to the whole team.

Insulating Tests from Interface Changes

I've been reading Robert Martin's Clean Code book recently and just finished the testing section yesterday. I've been TDDing for a few years now and am a big fan of using libraries that help make tests more readable, for example hamcrest matchers but I wasn't sure if the arguments put forward in the Clean Code book for building up a DSL for testing sounded like a step too far. I thought I'd give it a go today. What I found was that not only does it make the tests easier to read, it also makes them much faster to write and far simpler to handle changes to the code they are exercising. Building up a test DSL insulates your tests from changes in the production code.

Consider the following interface:

List fetchRecentEventsForPlace( EventsForPlaceSearch searchData );

Here's the old style test that I started the day with:

@Test
public void recentEventsCanBeLimitedByArrival_old() {
  Plan missedArrival = planFactory.missedArrival(FIRST_JAN_09);
  Plan missedDeparture = planFactory.missedDeparture(FIRST_JAN_09);
  persist(missedArrival, missedDeparture);

  EventsForPlaceSearch search = new EventsForPlaceSearch(place)
      .withEventTypes(EventType.ARRIVAL)
      .withPlanned()
      .withUnplanned();
  List results = planRepository.fetchRecentEventsForPlace(search);

  assertThat(results.size(), is(1));
  resultsHaveEvents(results, missedArrival.getEvents());
  resultsDoNotHaveEvents(results, missedDeparture.getEvents());
}

As you can see, I had already started building utility classes to help construct test data, making the tests easier to read and write. But when it comes to executing the code I want to test, I am calling the production code directly in the test:

EventsForPlaceSearch search = new EventsForPlaceSearch(place)
  .withEventTypes(EventType.ARRIVAL)
  .withPlanned()
  .withUnplanned();
List results = planRepository.fetchRecentEventsForPlace(search);

The next example shows the refactored test, using more literate function calls to construct the EventsForPlaceSearch object and perform the search:

@Test
public void recentEventsCanBeLimitedByArrival() {
  Plan missedArrival = planFactory.missedArrival(FIRST_JAN_09);
  Plan missedDeparture = planFactory.missedDeparture(FIRST_JAN_09);
  persist(missedArrival, missedDeparture);

  List results = recent(arrivals());

  assertThat(results.size(), is(1));
  resultsHaveEvents(results, missedArrival.getEvents());
  resultsDoNotHaveEvents(results, missedDeparture.getEvents());
}

private List recent( EventsForPlaceSearch search ) {
  return planRepository.fetchRecentEventsForPlace(search);
}

The immediate benefit of this change is the literacy of the test, you can quite easily read that the test is fetching all recent arrivals. The second benefit that you notice when adding subsequent tests is that suddenly everything is faster to write:

recent(arrivals());
recent(departures());
recent(plannedDepartures());

As if this wasn't enough, the other benefit is that the tests are insulated from changes to the production code. This did not occur to me until I actually had to change the interface of the fetchRecentEventsForPlace function while developing it. When I came to integrate the first cut of the function into the rest of the application I realised that most places it was used would require a slightly different data structure. Rather than forcing the application to perform the conversion it made more sense to change the interface of the repository to be aligned with the requirements of the client. I ended up changing the return type from List to SearchResults:

SearchResults fetchRecentEventsForPlace( EventsForPlaceSearch searchData );

Now normally a change like this on some well tested code is a pain, because you have to go to all the tests and change the expected return type and then change the expectations of the test to work with the new return type. This time however there was only one place to change, the method that provides my test DSL. To accommodate the change I simply modified the one function like so:

private List recent( EventsForPlaceSearch search ) {
  return planRepository.fetchRecentEventsForPlace(search).getRows();
}

I did not have to change one line of my test methods and all my tests are still valid and pass. This is the way it should be. Thank you Uncle Bob!

Grails Book Published

For the past nine months, I have been writing a book on web application development using the Grails framework. For anyone who doesn't know, Grails is a web application development framework that uses the Groovy programming language and a convention over configuration approach popularised by Ruby on Rails.

This book is not intended to be a complete reference to Grails, that already exists. Instead, this book takes a step by step approach to building a useful web application in Grails, introducing relevant and interesting aspects of Grails along the way. The aim is to show off some of the great features of Grails that allow Java web application developers to build slick web applications quickly, while leveraging their existing knowledge of the Java platform. While building the application I discuss some of the design decisions that I have made and also point to resources for further investigation of Grails features where needed.

The book (Grails 1.1 Web Application Development) is available from Packt Publishing, and you can try before you buy with a sample chapter. The book contains the following chapters:

Chapter 1 presents a short state of the nation of Java web development and makes the case for a framework like Grails. At the end of the chapter we will install and create a Grails project.

Chapter 2 covers the use of Grails scaffolding to generate some simple pages to manage users and roles for the application.

Chapter 3 shows how to post messages, where we write the first basic functionality for the application by allowing users to post messages that can be shared with other users. This chapter introduces a number of basic concepts for Grails development including: controllers, validation, Groovy Server Pages (GSP) and Grails Object-Relational Mapping (GORM).

Chapter 4 covers an introduction to Groovy. Here we take a short break from the Grails framework to get a better understanding of the Groovy programming language. We will cover just enough of the language to be able to proceed through the rest of the book.

Chapter 5 shows how to use our first external plug-in to add authentication and authorization to the application.

Chapter 6 covers testing, where we introduce the different levels of automated testing that are available in the Grails framework. We see how to write unit tests with new support for testing in Grails 1.1. We also cover integration tests, and install a functional testing plug-in.

Chapter 7 covers file sharing, where we allow users to share files through the application by introducing file uploads.

Chapter 8 covers some advanced querying techniques, using Hibernate criteria support in GORM, to implement file version history.

Chapter 9 introduces Grails services in more depth. We see how to extract logic from our controllers into services to keep the application maintainable.

Chapter 10 introduces more advanced GORM techniques, such as persisting inheritance and performing polymorphic queries to enable tagging. We also delve into GSP a bit more by using templates to encapsulate view components.

Chapter 11 covers AJAX and RIA Frameworks – Where we improve the user experience with AJAX to allow users to edit tags in-line and use the RichUI plug-in to create tag clouds and perform auto suggestion when editing tags.

Chapter 12 shows us how to use the Searchable plug-in to add a search feature to our site in a matter of minutes. We also provide an RSS feed and a REST based API for managing messages.

Chapter 13 show us how to build our own plug-in, where we follow the example of the Grails plug-in community and extract our tagging code into a plug-in that we can use on future projects.

Chapter 14 shows how to package and deploy the application ready for use in a production environment. We then discuss some next steps that may be worth investigating to handle real world situations.