Episode 188: Requirements in Agile Projects

I found this to be one of the best episode. The clear difference between agile and Agile coupled with a frank admission that Agile has gotten a lot of stuff wrong was refreshing. I believe the mentoring style was appropriate because rule following zealots have managed to make Agile into a process less agile than traditional waterfall by ‘it must be done this way’ and ‘we must follow the process’. A gentle, clear and firm explanation of why this is completely broken makes this podcast an invaluable digital asset.

I found this a provocative discussion, in which most cases I fully agreed with the comments made.

As a discipline, “we” will continue to benefit from mentoring like this as long as sone big-A Agile teams don’t “get it” (or “grok” or otherwise grasp) that a “shared mental model” is crucial for all but only the smallest of software development efforts.

In my experience, we still have pseudo-agile developers who care only about the “shared mental model” that resides only between their ears. If it ain’t disseminated, it ain’t agile 🙂

Keep up the good work!

I’d describe the different flavors of agile like this:

* “Big A” agile – the approach to software development promoted by the agile manifesto and espoused by people like Ward Cunningham, Martin Fowler, Kent Beck, Bob Martin, etc..

* “Little a” agile – the varied, eclectic practices that descend from o intersect with “Big A” agile (e.g., Kanban, SCRUM, or some roll-your-own hybrid).

* Degenerate agile – slavish adherence to selected agile practices without an understanding of their purposes and goals.

For some reason, throughout this broadcast, James Robertson refers to degenerate agile practices as “Big A” agile, which is incomprehensible, misleading and dangerous. It’s likely to drive people away from “Big A” agile writers (Beck, Fowler, et. al.) – and those people are still the best guide (by far) to using software to create business value.

Indeed a disappointing episode. So requirements do not change, only our understanding of them does? The XP ideas came out of Webshops where the requirements *did* change regularly. You can blame the founders of those shops that they did not “think their business ideas through”, but that’s pointless: The very point of agility is responding to change as opposed to following a plan (remember the flickr story? so maybe the online game they originally planned to implement failed, but the photo sharing stuff did work out. Do you want to blame flickr for not “thinking the online gaming story through”?).

And speaking of games: I wonder what the games industry (which is bigger than the film industry btw) will think of the comment that getting such a product right is much more trivial than getting “serious business software” right? We may or may not like the products of this industry, but as developers we should at least appreciate the fact that this industry makes some the most non-trivial software systems known to man (NASA and military stuff notwithstanding).

I enjoyed this episode – it was challenging yet pleasantly presented. It was really enlightening to hear their statements about documentation as artifacts on the wall providing the “why” rationale, and code being the documentation of “how”. Tests often document ‘what’, but ideally should encode the “why”s in their messages and utility method names for posterity after the original artifacts are forgotten. Keeping the “why”s visible is the key to business success of most software projects, not just technical success (Kent Beck says “there’s no such thing” – it’s a euphemism for failure). Obviously this is the hard part of software engineering (not least because developers don’t always meet end-users), and most frustrating when adopting agile and finding it hard to slow down and think about how to make the code reflect layers of domain concepts and the tests reflect the reasons. It evidently takes years to get good at this, but it sounds worth it.

@Jeff Sternal I agree. However I think James is focused on “A”gile of the past as it was unintentionally adopted by pressured and exploitative managers. “A”gile was always, as Suzanne says, “a terrific idea”.

James rightly said focusing on speed is wrong and doesn’t “necessarily produce business value” – speed doesn’t have a directional component. Velocity on the other hand is about capacity to provide value, and doesn’t only span the narrow “A”gile realm of development – real “a”gile is about the whole value chain from gathering pure observation data all the way through to support and decommissioning, and realising value quicker by realising failure quicker. Ideally your first iterations are in the right direction, but even if you fail and change direction, your speed (purely magnitudinal) can independently help you estimate future velocity, assuming you released anything at all and your code is flexible and (to take the metaphor to the extreme) won’t increase ‘drag’ resistance over time. These scary assumptions are what make “a”gile practitioners promote /short iterations/ (duration not speed) for the ‘invisible’ development process /after features have been planned/.

In his talks, Kent Beck makes no apologies for the /values/ of simplicity and feedback in Extreme Programming (XP) but honestly admits that his “A”gile /practices/ are only his current thoughts on how to achieve a steady flow of features – i.e. they are the baseline for a continuous improvement /process/. As well as being critised for being too “quick”, he is also unpopular for experimenting with small ‘slow’ iterations at the refactoring level. He finds it uncomfortable having someone watch over his shoulder, but he selflessly lets them. He’s done more than anyone to think, document and open up about his current practices so that others can do better.

As another agile manifesto signatory, Jeff Sutherland also endorses the /values/, but clarifies that the continuous improvement process is where the power lies (“We /are uncovering/ better ways”…), not in any set of /practices/. James rightly says requirements don’t change, so agile is not about keeping up with business change. He’s addressing a common misunderstanding, not how Jeff actually sees it – it’s to drive business change forward as you fulfil requirements after fewer iterations (not one), and if you follow some sort of code review practice (not necessarily pairing or TDD), you can then fulfil new requirements with the same codebase if they arise. What’s unique about “a”gile is that after 7 years of honing, Jeff can demonstrate that XP and Scrum /practices/ within the Scrum /process/ are /already/ very effective because the continuous improvement process can give him that data!

Back to documentation, James rightly says documenting “why” is important, but when he says you “need” up-front documentation for outsourcing providers I think he means you are currently forced to. However Jeff points out that outsourced teams in India and China are more and more becoming “a”gile, which James may not be aware of.

I think Jeff Patton, in his QCon 2012 talk “Co-Making Great Products”, explains well that agile processes don’t ensure success, they just *explain* /current/ baseline practices which then *motivates* people and in turn facilitates *relationships* where /repeated/ failure is unlikely. In discussing User Experience design, he also gets at the same key issues (with examples) that the Robertsons call “systems thinking”: obtaining shared understanding, not just shared documents, through agile communication and discovery – making hypotheses about what will produce business value, telling and discussing user story maps with the customer and thus avoiding sub-optimising a development silo in your value chain.