Jeff Patton opened his presentation with some advice on giving speeches: start late, finish early and don't discuss anything you don't understand. True to this advice the session started a little late, but proved to be well worth the wait.

His premise was that while XP teaches you to question your practices we nevertheless seem to be moving towards a unified agile process - process rigour or rigour mortis? He conceded he was unsure, but that it left him jittery. And he then expanded with an example: accompanied by Pink Floyd, he introduced Roger, a PO who believes he gets this agile thing - each story is just another brick in the wall on the path to completion. He needs to deliver a complete feature set to a client on a given date. Luckily, he works with Melanie, a developer who is confident that they can deliver.

All starts well, with the burn-down chart tracking a smooth line towards completion, until a few iterations in where problems are discovered. No problem - new stories are placed on the backlog to address these. Suddenly, though, there are more stories but the same velocity - it's going to take longer to complete.

Roger has made a common mistake and confused iteration and incrementing. Jeff referred here to Fred Brook's No Silver Bullet and the concept of essential vs. accidental complexity. Essential complexity is the werewolf - there's no way to kill it and you need to learn to live with it. But determining that essential complexity is hard, perhaps the hardest part of software development.

Jeff also emphasised that the snowman model does not help in understanding this. The model shows each iteration as producing a potentially shippable product increment. Unfortunately, potentially shippable means different things to different people. He suggests instead extended the snowman to a four level construct - a product is made up of multiple releases, each of which contains multiple iterations, which are in turn made up of multiple stories.

An incremental model lets you build the solution piece by piece. The problem is that you may be building the wrong thing. Iterative processes allow you to grope your way towards the desired outcome, starting with a light sketch and filling in the details. However, this creates a new problem: how do you deal with the uncertainty of not knowing what you'll get, especially in an environment where a software release may just a pre-requisite for effort elsewhere in the company?

Jeff suggests three solutions:

Focus on business value

Software is built to serve business objectives. Instead of cutting stories when you find your date moving out, remove business goals. One business goal will map to an entire set of stories which rarely work in part. Software that only completes a section of all your goals is useless, whereas software that solves a subset of goals completely adds immediate value.

Don't choose the solution too early

The are many ways to crack a nut. There's no need to decide if you want a Porsche or a Mini immediately - instead, scope both solutions and choose those that will best fit your priorities and timeline.

Build the base and then polish

You can't build a bus without wheels, even if the engine is perfect. So start with the minimum and build an end-to-end system. Then revisit and add features, robustness, security, performance and polish. A system the customer can see working allows immediate feedback and, unlike modules built in isolation, can be delivered if required. It's better to deliver a system missing features than some perfect modules you cannot use. And there's even less point in building these features if you're not yet sure you're building the right thing.

Don't know what I want but I know how to get it

His closing thought echoed Johnny Rotten - we don't know what we want, but we know how to get there. Don't run agile projects like waterfall - not every iteration will produce high quality, shippable software. But you might just end up building the right thing instead.

Tuesday afternoon was polished off with the intriguingly titled 'XP: My Greatest Misses 2000-2008'. Presented by J. B. Rainsberger, the founder of XPDay North America, it documented some of what he considered to have been his biggest mistakes since his conversion to the XP camp in 2000.

He came to XP while working at IBM, in an attempt to avoid defect ping-pong with the testers. After discovering XP he became an instant and enthusiastic convert, literally yelling its praises in the hallways. Finding he had failed to convert any of his colleagues he instead adopted a stealth strategy - demonstrating the qualities of the methodology by the quality of his work, confident that his continually low defect count would win him enquiries as to his methods. Instead, he found he left people pissed off, feeling they had been shown up by this upstart. So he tried a third method - proposing himself as a coach, only to find that his credibility was shot and that without credibility there are few who will listen.

However, help was soon at hand in the form of Toyota, who were starting the first XP project in Toronto. With a plan of building a base from which to expand to other engagements J. B. and his cohorts spent a lot of time massaging data and ensuring that their numbers were the best possible. Unfortunately, they found velocity was not a speedometer, and the impressive numbers did not equate to software. Management soon imposed their will and a quick exit was effected upon the cancellation of pair programming.

Later, he found a great opportunity to learn to teach when offered a role as a tutor for Java topics. Not having the emotional investment left him free to concentrate on learning to teach, rather than on the topic being taught. Eventually leaving to start work on a book, he began consulting, with the intent of whipping developers into shape and letting the agile concepts flow outwards into the organisation. On his first attempt he found a receptive team and started them anew with the aim of getting something that could be run in short order. However, velocity was low and so the stories were split into ever smaller segments. This continued under a BA estimated the number of stories required to complete the software at a notable 12000. As results had failed to materialise he had focused on a single problem, leaving the bigger picture at risk, saved only by this timely intervention.

Later he found himself in a similar engagement, introducing a team to agile, only to find that while some members of the team were enthusiastic in their engagement others insisted on withdrawal, marking out their personal space and avoiding communication with the team. Further, each team was being subjected to multiple trainers, each with an individual approach and style, and yet none thought to ask the team - what;s going on, what sucks? In this case an emergency retrospective let the discussion flow and sorted the problem, but without a view of what's going on within the team you're likely doomed.

He then briefly addressed motivation. People are motivated to do something to the extent they believe they can do it; to the point they can see the consequence of doing it and where they wish those consequences to occur. Where you have motivation problems, these three principles are a great place to start.

He finished with a flash of insight: while waiting for a flight in the barren wastes of Pearson airport he had a flash of insight, suddenly realising he had screwed up every previous agile engagement. Immediately he put pen to paper and wrote an open letter of apology, later published as the Last Word in Better Software Magazine. At the end of the day, it's not technical skills that make the difference but people skills.

Johanna Rothman, author of Hiring the Best Knowledge Workers, Techies and Nerds, was responsible for an interesting session on Tuesday afternoon. Her topic was Hiring for an Agile Team, and her premise was that it is primarily people, not skills, that make your agile team successful.

She differentiated hiring for a traditional team by stating that differences between people are more pronounced in agile teams, as successful teams tend to be more collaborative, cross-functional and indulge in intense focus while blurring traditional team roles. This creates a need for new team members to be flexible, take initiative and respect and like each other to the extent required by close week on a daily basis.

She therefore focuses on the people side of hiring - as she said, the technical side has been done to death. However, unlike the titans of the industry such as Google and Microsoft she actively discouraged riddles and other abstract approaches. Instead, she proposed the use of two primary methods: auditions and behaviour description questions.

Auditions are any method of viewing the candidates work first hand. This could encompass their solving a business problem, working with the team in an exercise or working together on the whiteboard. Instead of relying on the expressed abilities of the candidate you get a first hand view of their actual approach.

Behaviour description questions are those horrible sticky question everyone loves to hate - “Give me an example of a time where you...” or “Tell me about a time when you...” - open ended questions that force the candidate to articulate an experience. Both the articulation and the experience chosen offer a lot of information, as long as you limit it to a recent experience - after all, people mature and change, and an experience several years old tells you little about current approach.

She also suggested a few other tools - closed questions to establish facts, perhaps leading into an open question; hypothetical questions to see how the candidate may react to certain circumstances; and meta-questions to draw the candidate into the interview and force a switch of perspective.

Along with riddles, she suggested that traditional questions such as 'why do you want to work here' are pointless - after all, few candidates are honest, and if they really want to work specifically for you you'll never need to ask that question. She also cautioned against non-work related questions, as they can easily run afoul of anti-discrimination laws.

Her closing thought was that a good interview should be like a conversation - finding a good cultural fit is a two way street, and each interview will demand different questions. And while planning never goes amiss, the interviewer must be as agile as the candidate to get the best out of the process.

After the keynote I retreated from the ballroom to a meeting room hidden in a corner of a mysterious mezzanine, where I heard Clinton Keith of Highmoon Studios talk about agile in the gaming industry.

Starting from a position of describing the evolution of the gaming industry from a producer of mass titles into one where budgets rival that of movies, he talked about how the industry had adopted waterfall methods as a saviour, and how his company found them lacking. On their inaugural project, Darkwatch, they found themselves with a generous benefactor and plentiful resources and yet still found themselves digging a deep hole. Realising that in the circumstances only they could be to blame for this state of affairs they started looking for a way out - and, in the process, discovered Scrum.

Adopting Scrum they found many of their problems were solved but many more created. In particular, they found that applying the methodology in an environment where developers were in the minority presented unique problems, and that the to deliver a single release that was 'fun' presented a major challenge. Unlike many agile teams they operate on a product cycle of 2-3 years, and face the constraint of delivering to an audience where reviewers have huge controls over sales, yet rarely map to their target audience. They also faced the constraint of needing to engage a producer and establish IP rights prior to development.

Their solution was to operate a staged development process, starting with pre-production that created a working framework on which the production phases could build. This allowed them to work out the challenges in the project - such as a new platform, engine and game type - and quickly have a product to expose to audience samples. However, their initial attempt at dividing the team into functional groups did not bear fruit, creating pipeline problems and frustrating the distribution of work given the large number of disciplines involved. Instead they adopted a core team model, with specialist teams building features and helping the core team in adopting these modules. And their created a hierarchy of product managers, managing varying modules as separate products in order to eliminate straight pipelining.

They also found that while the developers found Scrum appealing, the artists considered it intrusive micro management. They addressed this by using Kandan and time-boxing - giving artists time boxes in which to complete a task, and tuning these time boxes to the desired solution, and creating a pipeline to address the various tasks in design and allow managed movement of tasks through the creative process. This offered a framework to identify waste in the process and by encouraging collaboration between the creative phases they found a significant amount of waste was eliminated.

One problem with the multiple product model was that build integration became a major impediment. To address this they assigned ownership of the build to the engine team, ensuring that someone was responsible for diagnosing build breakages and ensuring they were fixed. Build breakages were not limited to local notification but broadcast to all desktops in the company, raising their profile. And tests were created to play the game and allow continuous integration to detect integration problems on a nightly basis.

He closed by suggesting some further optimisations to the industry, in particular highlighting episodic gaming as a way for the industry to lower risk and allow a more iterative interaction with consumers. He also emphasised strongly the need to apply methodologies to the industry - while the principles remain sound the industry is unlikely to adapt itself to the theoretical methodology. Instead, adopting a pragmatic approach allows one to realise the benefits of the approach in a wide variety of businesses.

Agile 2008 opened at the somewhat startling time of 8:30 with a keynote by journalist James Surowiecki, author of The Wisdom of Crowds. He opened with an anecdote about the experiences of Francis Galton, famed both for statistical science and infamous as the coiner of the word 'eugenics', in relation to the surprising accuracy of the averaged guesses resulting from a fair guessing game as to the weight of an ox. And he then built upon this premise: that, under the right circumstances, the composite wisdom of a group will usually exceed the estimates of experts in the field.

He continued by backing this up with both parable and statistics. Guessing the number of jellybeans in a jar is a favourite fair game, and his claim was that over three or four attempts the average guess of the participants will outperform any individual within the group. He continued with 'Who Wants to be a Millionaire', claiming that while nominated experts will offer the correct answer two thirds of the time, polling the audience produces a correct result an impressive 91% of the time. Among other examples he referenced were Google's Pagerank - which determines the importance of a link by a count of the references to said resource - and NASA's ClickWorkers, a tool they created to use interested parties to classify craters and which produces an average accuracy equal to that of a trained geologist.

The important words are, as always, under the circumstances. After all, and as he commented, we are accustomed to consider a mob to have the lowest IQ of its constituents. And events too often prove us right on this score - from lynch mobs to panicked stockbrokers amplifying a crash we are frequently reminded that crowds will often show a surprising lack of wisdom.

He described these circumstances as requiring three preconditions. Firstly, you must be able to aggregate the judgements of the group. Secondly, the group requires diversity. This was comprised of several elements - apart from the obvious cognitive diversity, the requirement to avoid confirming a pre-chosen conclusion and avoiding groupthink came to the fore. Finally, one must ensure that the decision is that of the group, not of a dominant agent within the group. In order to benefit the outcome must be the aggregate of the individual outcomes within the group, rather than a uniform outcome created by peer pressure and a desire to conform.

His presentation was certainly compelling and communicated interesting information, if not desperately original. After all, such concepts have been at work in prediction markets for some time, and their success proves the concept much more effectively than a collection of theory. Nevertheless, bringing such concepts to the attention of an audience perhaps less exposed to such implementations can be naught but good, and it certainly provided a solid and polished opening to the event.

For my sins, I find myself sitting in a hotel in Toronto, watching the rain steam down the windows. To be fair, I could have done this at home in London, a city in which clement weather is but a month. Instead I've been shipped 8 hours west, across the Atlantic and dangerously close to the US in order to attend Agile 2008, an agile software development conference who this year has ventured beyond the borders of the land of the free and into the dangerous northern wastes.

The northern wastes, mind, I take on hearsay. Since arriving yesterday afternoon I've ventured from the hotel only to get dinner, and even then - thanks to the rain - returned via the safety of the subterranean PATH network. Rumour has it that beyond these confines lies a city, perhaps even a country, yet until some sunshine and a gap in the conference schedule illuminates these it would be imprudent to assume their existence.

And so here I am at Agile 2008. Four days of talks, tutorials, workshops and buffets on the subject of agile software development - and any tangents the presenters can draw from such. One day down at this point and I appear to have had the best of the three of us here - Ivan found the crass commercialism of the keynote a terrible start to the day, and Gojko has sworn to avoid all tutorials and workshops for the remainder of the conference. So soon this will be followed with some details on the material I have seen; tonight, however, my jet-lag forestalls such exertions, not to mention the lack of WiFi in what passes for hotel rooms in these parts.

So, intrepid reader, stay tuned and - assuming I am not distracted by Serbians bearing drinks - I shall get some write-ups up here as soon as practical. And should the worst come to worst, it's something to do on the flight home.

Having spent all too many hours enjoying Air New Zealand's hospitality - in the loosest sense of the word - I remain perplexed as to the design of their entertainment system. Things have moved on a lot since I last did a long-haul trip in 2005 - while we were chuffed then to have multiple movies broadcast to your seat, we now have personal on-demand systems in each seat. There's just one problem - they're somewhat rubbish.

So rubbish in fact that Air New Zealand warn customers: Please do not press the buttons too fast, or your screen will freeze and you will be unable to use it for 15 minutes while it reset. Robust, that. And the temptation is definitely there - there's a notable pause as you press each button, and sometimes it misses them all together. It doesn't help either that when you try to select a movie you have three fifths of the screen occupied by a large blurb teaching your grandmother to suck eggs, and therefore have a minimal section of the screen in which to page through said content. Streaming is also somewhat slow - the movies are compressed enough that MPEG artefacts are very clear, and you attempt to fast forward at your own risk.

The most vexing thing about all of this is that these problems have been solved: either you have smart terminals style devices talking to a central server (there's a hell of a lot of embedded hardware that can decode MPEG very cheaply) or you use dumb terminals with a multi-user server driving everything. How you end with with terminals that offer shocking performance and a 15 minute reboot time is beyond me.

It's a telling reflection on the state of software engineering that not only is this kind of half-arsed solution produced these days, but that it's both accepted and lauded by the customers and users. We wouldn't accept this from an engineer of any other form - why here?

The magic of hypertext was the ability to suddenly make a glorious, if somewhat unstructured, whole from a mountain of disconnected data. And so we end up with the web, and the ability to lose hours bouncing in to Wikipedia and ending up God knows where.

This is a much nicer approach than the old world, with its disconnected computers and systems, each man being but an island. However, I can't but notice a nasty slide back to this world, only this time we'll be stranded on different platforms.

The much-hyped 'Web 2.0' is the worst offender. Google Reader for instance - fantastic application, but your data ends up in a silo. Newsgator have a better idea here with the ability to synchronise data, but lack a Linux client and have but a rather basic web client. And the social networking sites take this to extremes - I have a presence on the web. Here. Why, should I want to join MySpace (heaven forfend) or Facebook (which seems suddenly very trendy), do I need to create another one?

Would a true networking site not allow me to draw my disparate services together: mix in my homepage feeds with my LinkedIn profile, my IMAP e-mail and my Flickr photos? Let me manage my contacts via iSync and track down my friends via this data source? Let me read my news via the best tool, whether I'm on the web, my computer or my phone?

There is some movement: Facebook have their new API but it is still rooted on their platform - you draw data in to them, not out. Flickr are fairly open, to their credit. Google, surprisingly, aren't - POP mail in GMail looses you your valuable filing metadata, and you are completely buggered with respect to Google Reader.

I guess in a way in all comes back to the Semantic Web - if you can exchange data freely then all sort of interesting things will pop up. While you silo it a huge amount of energy is wasted on duplication. And that's bad for all involved.

(We apologise for the sloppy writing in this post. It's that sort of day...)

Oh my. Someone went and leaked a HD-DVD key on Digg (far from the first place it ha appeared). The world's worst piece of legislation is invoked (DMCA) and they get a take down notice. They try to implement, only to surrender to an overwhelming flood. Indeed, you can read about nothing else on Digg at present.

What continues to amaze is that the MPAA and the RIAA and the like continue to believe that they can make DRM impenetrable. HD-DVD is still a niche product and yet it has been subverted. It's somewhat concerning that they think they have a right to a string of hexadecimal digits. It's even more concerning that the US government is behind them all the way. And it's just plain daft to believe they can put this genie back in the bottle with a few cease-and-desist notices.

Anyway, just to get in the game: 09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-c0. Hurray for the UK - surveillance society, but not ruled by the media companies (yet, anyway, although the EU Copyright Directive is bad enough).

Easter is normally a time for devouring large amounts of chocolate and avoiding the overcrowded roads and tourist spots (i.e. London). However, this year we're only two days in and I've managed to spend them quite profitably.

Along with a decent amount of Wind Waker, I've also added in an exception class-loader and third-party checker support for CheckStyle-IDEA. The changes are in SVN now, just testing to go - which is the bit everyone hates. Still, it does give me a lovely righteous feeling.

Now for more cream eggs...