Nobody cares about your tickets

As engineers we like to speak in tickets and stories and bugs IDs etc - those units of currency we find easy to grok; they're usually finite, easy to define, narrowly scoped, and uniquely identifiable.

Our customers - for the most part - like to speak in outcomes and business plans and commercial objectives and projects.  The longer term, more vague, abstracted-from-the-work things that we like to digest by breaking them down into those stories etc that we're so comfortable with.

This can lead to a weird situation where you have all your iterations scoped out and your releases forecast (with painstaking detail in every story) yet the business still lacks clarity about what's being delivered by who and when.  Hmmm.

I think a bit of generalisation can help us to understand this.  Lets appreciate that engineers, for the most part, are deeply practical people who are comfortable handling actionable tasks.  Since we're also geeks - again for the most part - this descends quickly into technical details.  You'd be surprised how unintelligible this can all be when you're not a developer.

What I've learned over the years is that stakeholders are interested in the meta problem, the commercial plans and outcomes, not the tangible problem, the technical tasks and code which need to come together to solve it.  As technical leaders we need to talk about the meta problems with the business, and the technical problems with the engineers.

This comes with two obvious challenges - whenever we're talking about the same things but using different currency we rely on a degree of interpretation and some things can be lost in translation.  This most commonly surfaces as that old "unclear requirements" thing.  The other challenge is making sure that, as we solve our technical problems, we're solving our meta problems.  This isn't easy and this is why great product managers are so crazy hard to find!

This kind of interpretation of work - the aggregation of agile units into bigger picture project plans - is usually an early casualty of agile adoptions.  It is reminiscent of waterfall so we label it bad.  But it is a helpful mechanism which bridges the gap between agile execution and regular business planning.  It gives you both currencies.

Get awesome product people - they are the only other ones who live in this gap with you and have to speak both languages.

Adding a feature dimension to planning boards

Planning boards – sometimes called task boards – are a great way to make work visible, facility teamwork, and surface impediments.  The work required for a given iteration of software is broken down into a collection of individually owned tasks and then, as work progresses, we watch those tasks migrate towards the right hand side with a sense of satisfaction.

This is unprecedented transparency and one of the reasons why I love it, but it does have one small shortcoming.  If you’re not intimately familiar with the work (say, for example, a non-technical stakeholder) then it can be a bit hard to determine meaning.  Yes things are moving along, but what really are those things?  Of the 4 stories we’re doing this sprint, which business feature does this progress (or impediment) relate to?  It looks almost all done but are we saying that we’ve done 3 of our features and we’re working on the 4th or that we have we done 80% of each of them?

We do want the business to be engaged and to participate in the process so we ought to try and make it interpretable for them.

Some of my guys recently started adding a 2nd dimension to their planning boards; features.  By adding a horizontal view which groups tasks by the requirement they relate to as well as the usual vertical view which groups tasks by their status (not started, done, etc) you get an at-a-glance view of the progress of each piece of business-relevant functionality, rather than just a broad sense of things happening.

Even I find this easier to consume!  Rock on you little geniuses (you know who you are).

MVP in the Enterprise

Minimum Viable Product (MVP) is a well understood concept in startups.  It is all about distilling your features down to the barest of essentials – those things which really make your product your product - in order to get something out rapidly and be able to quickly iterate guided by feedback.  From working at both ends of the startup-to-mature-business spectrum, it occurs to me that the enterprise could learn a thing or two from its lighter-weight cousins.

To broadly generalise the circumstances I’ve seen:

We’re pretty good at this MVP thing in early stage startups.  Mostly because, as we plan an iteration, we are fully aware that we might not be around for another iteration so whatever goes into this one has to count (in fact it might count for everything).  We think lean.  We’re typically good at serving our customer’s most immediate and pressing needs.

In the enterprise we have the luxury of working with the confidence that, even if the immediate next set of features don’t quite hit the sweet spot, we’ll be around for many more iterations yet before times start to get tough.  We think deep and wide.  We’re typically good at serving a big strategic master plan.

Being able to take a longer term view of a roadmap is definitely an advantage and having to spend time worrying about things like scalability is a nice problem to have, but this doesn’t mean that we can’t also think lean.  In fact some of the most value I’ve added has been in the amount of work I haven’t done...

It can work in big business.

Recently I looked at a system to capture and analyse certain customer activity in order to get an earlier prediction of lifetime value and make data-driven cross selling decisions (which ought to push up average revenue a notch) and, on the surface, that seems like it’s worth the pretty hefty licensing, integration project, and infrastructure costs.  We started off modelling it and running the numbers; at first synthetically and later with real data.  What we learned through this early prototyping is that those features which were initially so attractive made very little difference to our business, however our little robot was uncannily good at identifying fraudulent activity.  So we switched the direction of the project, chose an alternative platform, saved a bunch of money and got some functionality which genuinely benefitted us.  Sometimes you know what the real benefits are going to be up front, sometimes you surprise yourself along the way.

We’re also planning to introduce a pay-as-you-go commercial model for our feed products.  When you add up a metering system to watch client consumption, a dynamic pricing system to set tariffs, a billing engine to produce the required invoices, and some reporting tools to keep an eye on its performance it starts to become a fairly significant undertaking.  Will customers like it?  Will it work like we hope (add top-up revenues to committed subscriptions) or like we fear (cannibalise commitment for shorter term hits)?  The best way forward in these circumstances is to ask what is the absolute minimum amount of work I can do to see if this works?  As it turns out we could do a some basic instrumentation (to get simplified metering), use some static pricing, and do the billing manually.  Net result?  A much smaller piece of work – useable in production with real customers – demonstrating how successful it’s going to be.  Now we can build on this with further iterations until we have the fully-featured and refined version we first dreamed up.

There are many more examples, some of which have worked out just fine – so we’ve kept building on them until they were feature complete, fully automated, and enterprise quality.  Others have failed – so we’ve cut our losses (fail fast fail cheap; writing off a few thousand not a few million) and moved onto something different which did turn out to be a winner.

Just because we have the budget and the appetite to do the whole shooting match right away doesn’t always mean we ought to.

The obvious risk here is becoming too short sighted with your plans.  The secret to making this work is to imagine big but plan small – keep a long term roadmap and have a clear vision of where you want all your products to get to, but take small, tightly scoped steps along that path.  Stop and evaluate frequently, adjust big picture where necessary, rinse and repeat.

Architecture in Agile

When a team first starts practicing agile a question that usually comes up fairly early is where architects fit in.  In a set of processes which usually considers ‘big-up-front requirements’ it’s nemesis, it can be a startling realisation that we (the engineering side) might have our own version of that.  Architecture tends to be associated with big-up-front specs and, if you’re not doing any of that big-up-front stuff, then where does the role of an architect fit in?

The first step towards finding your answer to that question is to understand what architecture actually is:

Architecture isn’t something separate and distinct from your projects that you can either choose to indulge in or choose to abstain from – architecture is simply the shape of what you produce.  Everything piece of software in the world has architecture, the only question is whether it was done by design or by accident.  The question is essentially whether you drive it explicitly and control your technical destiny or whether you leave it to chance and hope for the best.  Driving it would be the job of the architect.

To make a slightly controversial statement, architecture by itself has no value.  But architecture, as defined as ‘the correct technology platform meeting the current and future functional and non-functional needs of the business’ is worth everything that you can put into it.  I believe in architecture.

Architecture must be explicitly handled – lest it get away from you and leave you with unsustainable systems – but that’s not the same as saying that you must have an architect.  You simply need to have the responsibility for the shape of the system (the technical decisions) clearly understood.  Why not devolve the responsibility?  Something common to most agile approaches is the empowerment of the teams to self-organise and take control of the destiny of their own product.  Architecture is no different and, where you can identify neat boundaries between different products owned by different teams, this type of ‘product architecture’ works very well with minimal enterprise oversight.  Less distinct separation between systems generally = more enterprise oversight.

Somewhat counter to the freshman agilist’s intuition, you do need to make some up-front decisions in most projects.  Agile is about making sure you know enough about something to be effective by when you need to know it – it’s not about knowing nothing.  The trick to handling technical designs is knowing what those up-front decisions should be and what’s better handled as part of how the project evolves.  Unfortunately this judgement takes some experience to develop, but a good guideline is to stick to things which can’t be easily changed later – such as schemas, APIs, and namespace.

If you’re a formally nominated architect in an agile environment my advice is stick to patterns and delegate implementation decisions.  When you are dealing with a flow of work and you can’t initially nail down the complete set of requirements in buildable detail – the business case for agile – then you have to let your vision for the product, and the size and nature of the commercial opportunity it supports, help you pick out the right patterns.  Patterns are the best currency for an architect to deal in if he has a team of smart developers keen to take some ownership; the ‘here is a design doc, build this’ approach becomes a constraint rather than a support.

In more larger or more complex agile environments – with multiple development teams and multiple products – an architects role moves into governance; the design of common elements and the way in which products integrate and interact.  What we might call enterprise architecture I guess.  The goal here is to put together a framework which promotes self-contained and distinct products, giving each team their own jurisdiction while still ensuring smooth overall performance of the whole estate.  In my experience this maxes out the total delivery throughput of the whole organisation and gives the satisfaction of ownership to the doers.

If you’re new to architecture (giving or receiving!) and need to make sense of all this then I’d recommend joining IASA because a key part of our mission there is bringing clarity to what an architect really does for a team, a product, and a business.

Let's Ban Agile

Bet that made you look.

Whenever I'm dealing with the wider business and I come across confusion about (or resistance to) agile practices I typically find that, once the few genuine issues are swept up, much of what's left comes down to semantics. And fair enough too - the extensive metadata that has built up around engineering processes is intimidating.

If we put all that aside and just talk about the basic practicalities of doing projects, you are pretty much just looking at an inventory of requirements that must be turned into an inventory of working product. To do that, distinct individuals need to execute specific tasks within certain time constraints (or, more colloquially, who does what by when). From there, we build up some rules to make sure everyone can cooperate effectively around those basics, and it is these rules we'd tend to call our development process/project methodology.

It isn't any harder than that. But, for some reason, we like to make it sound difficult through a plethora of technical terminology.

Instead of iterations, why don't we just talk about tackling big projects a bit at a time? [In fact, I'm yet to discover a better way of solving huge problems than breaking them down into a collection of smaller ones] Instead of backlogs, let's just have a prioritised list of requirements. Instead of standups, let's just have regular and frequent meetings to keep everyone on the same page. Instead of sprint planning sessions why don't we all get together to iron out a detailed plan before we commit to our next chunk of work?

I guess what I really want to ban is unnecessary jargon - at least when dealing with the non-technical contingent of the business. I think we should use common everyday language to describe what are, ultimately, practical and common sense activities.

Just because we're solving complex and difficult problems doesn't mean we need to use complex and difficult processes - in fact, one could make a case for quite the opposite.

Reporting & Agile

When I talk to people about their experiences with agile development I frequently hear - from the business side - that once all the bureaucracy has disappeared (which no one tends to see as a bad thing), they struggle a bit with the visibility of projects and the reporting of progress.

This view is usually 100% in contradiction with what I hear from the same organisation's development practice - who will usually tell me that visibility has never been greater and that they have an unprecedented degree of transparency and openness with their business partners.

Can they both be right?

When we, as engineers, make statements like the above we're usually referring to the public nature of our SCRUM/XP/etc artifacts - product backlogs and burndown charts are open to anyone in real time, requirements, estimates, and plans tend to be kept in readily accessible web tools, and of course most events in the process (such as daily standups and each iteration's demo) are public forums. What else do you need to have the most detailed, accurate, up to date view of every piece of work anytime you want it?

This is all true, but where we tend to go wrong is in underestimating the underlying complexity of something that inherently makes sense to us. There is actually a significant degree of interpretation - based on technical knowledge built up over years - necessary to answer a simple question such as "how are things going?" through agile artifacts alone; we just tend to be pretty good at doing it in our heads as we go.

Here's a simple example to underline my point. O2 has a pretty nifty self-service portal, and I can use it at any time of the day or night to see what my current mobile phone bill is and how far through my monthly quota of data and text messages I am. It works well for me because they have taken some underlying data about my account, put it through some pre-processing to provide some plain-English conclusions, and presented it on the web conveniently formatted so that I can interpret its meaning. Instead, imagine if they did transparency and self-service as we sometimes do in software delivery and just told me that all the data is there - the same artifacts that they use internally are open to me - and I must draw my own conclusions from some raw material. I guess I would install toad, connect up to their DB, select my rows from the billing table, join that with my rows from the metering table, and dump them into a view that showed me a current bill value along with my unused quota of texts etc. A few of us out there would probably quite like this alternative but, nonetheless, we have to agree that it is unlikely to be a level of service that the majority of O2's customers are going to be happy with let alone know how to use.

So what can we do differently?

Try to aggregate and simplify a bit for your stakeholders; provide just a little more interpretation and presentation on top of the solid data you already keep and it will be much easier for non-techies to make sense of. As long as you are not misrepresenting the underlying facts (easy to do unintentionally whenever summarising data for consumption by others) you'll probably find that your business partners will refer to that as the 'transparency' you've been trying to achieve and they'll thank you dearly for it.

Gates and Nonblocking Delivery Practices

A few weeks ago one of my guys sent me this article from The Agile Executive which is - although not explicitly said - about the collision between ITIL infrastructure governance and agile development methodologies. I use the word 'collision' very deliberately because those guys deserve credit for how well they've managed to knit those (often opposing) worlds together. Which brings me nicely onto gates.

Gates, defined as the "You Shall Not Pass" checkpoints which must be navigated throughout a project, are not inherently bad. They just get misused in the same way that anything we do has the potential to be applied over zealously.


Gates have a place in delivery and fulfill an important function. They serve as a kind of final quality checklist to make sure that all those things you said mattered and had to be done actually have been done. As long as they are few, important, and near the end of the development cycle, you can make this work.

Where this tends to go wrong is when gates are established without clear and measurable clearance criteria defined up front. What you have then is an obstacle with a variable and subjective success criteria - and then everyone wonders why a project arrives at the gate and struggles to meet the requirements to move on. An understanding of exactly what's on the checklists a project will face as it goes live and exactly how that will be measured has to be front loaded, so that development teams know exactly how to make a piece of work transition through the lifecycle smoothly and organize resources in advance.

If this isn't established early and you get a project logjam downstream then you often end up having to compromise - and that means compromising on key quality or operational requirements that you believed are important enough to gate - in order to restart the flow. The good old fashioned 'defects fixed later cost more' curve still applies here.

Another common misuse of gates is introducing them into a process primarily for reporting purposes. It is true that a regular set of gates established throughout the end-to-end project process provides a set of convenient handholds against which to benchmark progress (we are at 'customer feedback on specification' or entering 'test run 3' etc) but there are a couple of downsides that go along with that.

Firstly, it tends to encourage over-reliance on artifacts such as documentation and reports rather than working software as a primary work output. Secondly, progress becomes measured by the clearance of gate after gate, step after step, and revisiting a previous step is seen as a step backwards. That lines up with an easy to describe liner view of the world, however with most complex projects in most organizations things are not that simple and progress often involves degrees of overlap and leapfrogging.

Standups vs Team Meetings

When teams first start to pick up SCRUM, there is a tendency to let stand ups replace regular team meetings. The risks here are that your stand up will elongate and get off topic because it is the only chance you get to talk about issues as a group, yet your team cohesion will still suffer because no matter how long you can string out a stand up in the morning it won't be long enough to table all the things you need to get through as a team.

A stand up is a project-oriented meeting, all about running the day to day work of the team, has a tightly fixed agenda, is time bound, and specifically focused on what's required for today.

A team meeting is a team-oriented meeting, all about maintaining the team, continuous improvement, future requirements, and focussed on bigger picture topics.

They are definitely two very different meetings with two very different purposes, and both of them are necessary for a team to run smooth projects and keep productivity and job satisfaction high.

I think you might have it backwards

The most common criticism I hear about agile is also the most common mistake teams make in their early adoption of iterative practices - not fixing any certainty.

One of the benefits of managing projects using, for example, SCRUM is that you're not burdened with having to define all the requirements up front and plan out all the tasks and times for all the resources for 6, 12, 18 months. Trying to precisely define everything a project needs up front and trying to plan for every dependency, holiday, resource issue, structural or strategic change over a long period of time proves again and again to be an impractical task.

Enter agile.

Suddenly we don't have to worry about the detail of every change and every movement over an entire year, and we're introducing a whole bunch of flexibility. Where it goes wrong is not fixing any certainty in the delivery process at all.

Agile is about certainty and detailed planning, just detailed planning over a period which is possible to predict. Who knows what'll happen over a year, but you at least have a fighting chance of planning out the next 2-4 weeks will some degree of accuracy. Beyond that you can keep plans and requirements fluid and high level, netting you the right balance of fixing doable deliverables and keeping flexibility.

Software delivery needs certainty to happen - you have to be able to rely on some requirements and some resources over some time span in order to make progress - and agile isn't about not having any, it's about having it on terms you can foresee.

The nuances of Planning Poker

I’m going to skip right over explaining what planning poker is and how it works - especially since there are so many simple and effective explanations out there - and assume that you’ve at least given it a bash.

What I want to address is the spirit in which group estimation happens.

Every potential task owner (read: qualified opinion) estimates each relevant task as it comes up, but the reason for this is commonly misinterpreted. The purpose is not to force the eventual task owner to reduce his or her estimate, or to adopt an estimate that others in the group ‘prefer’ and nor is it a criteria for selecting an owner for a task, based on the estimate the product owner or scrum master likes the most.

The purpose is to draw out knowledge from the outliers, so that the whole team can benefit from their experience. When someone in the group pulls a card miles away from the rest (assuming a consistent understanding of the task) that’s something to explore. You might have a much higher estimate – perhaps that person has been burnt before by that same inconspicuous looking requirement masking significant complexity. You might have a much lower estimate – perhaps that person knows a good work saving shortcut or pattern uncovered during a previous project.

If you take the time to ask those questions and consider that information, the group learns from one another and everyone gets better at estimating.

And whose estimate is used in the sprint? The person who is going to be doing the work...

Don't out-specify your knowledge

Combining some points from this post by Joe Armstrong and this post by Dan Creswell gives rise to some interesting implementation thoughts.

Standardisation is important, particularly in the context that Armstrong frames it in - if the interaction between a client and a server closely adheres to a predetermined protocol, then the systems overall behaviour becomes more predictable and easier to adapt and fault find.

But then this depends on the maturity of your understanding of your problem and your solution. As Creswell says at the beginning of his post, defining too much up front can be a trap. Coming up with a set of standards that are too rigid too early might start you down a road of building that system instead of building the right system.

Standards, patterns, and contracts are all important - but timing is key.

Why are they never happy?

This is another one of those stories that my girlfriend will tell you explains why we don't get invited around for dinner more than once, but I'll tell you is dead essential to the world of technology becoming a better place - one manager at a time...

Anyway, I'm listening to this buddy complain about how busy he is, how hard he's working, how many hours his team are putting in, and how much stuff they're getting out - yet still his customers aren't happy. They're always finding something to complain about, pointing out the things that are missing rather than thanking him for the things that are there.

When I hear this, what I think most people are really saying is "my team and I are working really hard and really long hours doing the things we want to do for this customer". Whenever I've seen this issue on the floor, it has almost always been because the guys were dedicated and hardworking, but being dedicated to and working hard on the wrong stuff.

Think about it like this; if I really want A, and you give me B, C, D, E, F, and G, then I might be impressed by how busy you've been and how much work you've done, but I still won't thank you for it. Yes, B through G were relevant to my department and they were things I wanted done - no argument there - but if A was the thing that meant the difference between getting that big contract or not, or making this quarters numbers or not, then I would have happily traded it all to have just A.

I'm not saying that there aren't ungrateful people in the world, but I am saying that until you've adopted a partnership-based, customer-centric view of how you deliver technology, you haven't earned the right to complain about them.

Look frequently and carefully at your priorities, and compare them against what your business is being driven to achieve - you never know - you might even end up doing less work and having happier customers.

Making Decisions

I thought I'd start with this as my first 'proper' post of the year, mainly because decision making isn't usually considered a skill in it's own right, despite being deceptively simple and so commonly done poorly.

Making a decision isn't a stand alone task, when you make one, you really do a small collection of things together. You make choice from some options, you make a plan and delegate it, and then you follow up and review. Most people think that once they've picked their option they're done, but like Drucker says, you haven't made a decision until you have an action plan - you've only made a choice.

So let's talk a little more about effective choosing, as that's the first hurdle to trip over. The behaviors you want to avoid here are essentially procrastination. How many times have you been staring an obvious course of action in the face (or been presented with a fantastic opportunity) but instead of making progress, you're taking bets on which is growing faster - your number of choices or the list of data points people want to see for each one! You have to reduce your number of options to a manageable amount (3 is a magic number if you're stuck for inspiration), and while you should never ignore great last minute ideas just for the sake of rules, most business don't fully appreciate the cost of inaction. The point marked "doing anything is better than nothing" creeps up a lot quicker than you think.

The next thing you have to worry about is the ballooning amount of information on each of your choices. Too many and you risk it being treated like a smorgasbord (can we have this from option A, and that from option B) or being paralyzed by the sheer volume of statistics and technicalities. Try removing things that are the same for all choices - for example, if A, B, and C are all similarly expensive, why compare cost? And how about things you don't feel strongly about? I don't know how many times I've seen stakeholders agonize over details like the difference in reusability between 2 choices in a decision, and then when asked where else they plan to use the technology, answer "nowhere". If it's an attribute you don't feel strongly about, then it's unlikely to influence your decision, so why clutter the process with it?

We've talked a lot about paring things down, and now here's something to add in. Don't forget that for every choice you're comparing you need to know what you don't get if you don't pick it. Most people are generally pretty good at stacking up the costs and benefits, but opportunity cost is often neglected.

Hurdle number 2 is action plans and delegation - the behavior you want to avoid here are those deja vu meetings where you all sit around and 'review' the decisions you made, wonder at the lack of progress and then agree to 'update next week' all over again. Once you've made your choice, don't let yourself off the hook until you've agreed who is actually going to do the work, and by when. Follow the usual rules on delegation - make sure whoever you're going to hold accountable clearly understands their obligations, is empowered to do the work, and has the available resources.

Don't forget to follow up. You spend your time on the things that are important to you, and you're not making unimportant decisions, are you? The things that you focus on will get done - keep track of what's happening, make major milestones public, and use your sponsorship to see it through. And once whatever it is has been delivered (or has crashed and burned), you're still not quite finished. Now comes the feedback loop - what did you learn? How can the experience improve the rest of the organization? What pitfalls can you now help others avoid?

And finally, you shouldn't be afraid to revisit your decisions and change your mind. You should be keeping an eye on your customers, your portfolio, and your team, and continually assessing what any changing circumstances may mean for you. That said, you must resist being flighty as you need to show commitment, but keep in mind that "because we said we would" isn't a business benefit to completing a project - it's a sleepwalker's justification for not being in touch with the market.

Tradeoffs - You're Doing It Wrong

Our time is always our scarcest resource, and most of the web companies I know are perpetually bristling with more ideas than they could ever manage to ship. We always wish we could do them all, but at the same time we know deep down that it isn't realistic. Under these circumstances, tradeoffs are an inevitable and perfectly normal part of life.

The secret to doing this well is in what things you weigh against each other when deciding how to spend the resources you have.

It's so easy to start down the slippery slope of trading features off against the underlying technologies that support them. Fall into this trap and you'll get a small surge of customer-facing things out quickly, but stay with the pattern (easy to do - as it does reward you well in the short term) and you'll soon find yourself paralyzed by technical debt; struggling against continuously increasing time to market and building on top of a shaky, unreliable platform.

Instead, try trading off a feature and it's supporting technology against another feature and it's supporting technology. This isn't a carte blanche ticket for building hopelessly over-provisioned systems - it's simply about setting yourself up in the most appropriate way for future requirements and scalability, rather than foregoing whatever platform work that would have been diligent and starting off behind the curve.

Trading features off against their underlying technology is not just short sighted, it's also unrealistic. You see, the fact is that all customer-facing features have to be maintained and supported anyway, no matter how badly we wish we could spend the time on other things. And right up front, when we first build each feature, we have the most control we're ever going to have over how difficult and expensive that's going to be for us.

This is a philosophy that has to make it all the way back to the start of the decision making process. When considering options, knowing the true cost of ownership is as vital as the revenue opportunity and the launch cost. Remember - you build something once, but you maintain it for life.

The Confidence-o-Meter

A while back we had a project that just seemed destined to face every type of adversity right from the outset (we've all had at least one). Being a new line of business for us, we didn't even have a customer team when work needed to begin! It was going downhill, and with strict regulatory deadlines to meet, we needed to get it back on track. Additional complications arose because the team was new, and as such were still gelling together as they tackled the work. Let's throw in an unusually high dosage of the usual incomplete specifications, changing requirements, and unclear ownership that regularly plague software projects and you have a recipe for a pretty epic train smash.

They say necessity is the mother of invention (Plato?) and there was certainly no shortage of necessity here, so we got on with some serious invention.

The Problem

We needed a way to bring the issues out in the open, in a way that a newly forming team can take ownership of them without accusation and defensiveness creeping in. More urgently still, we had conflicting views on the status of each of the workstreams.

It seemed sensible to start by simply getting in the same room and getting on the same page. To give the time some basic structure, we borrowed some concepts from the planning poker process. There were some basic ideas that made sense for us - getting the team together around a problem, gathering opinion in a way that prevents strong personalities dominating the outcome, and using outliers to zero in on hidden information. As an added bonus, the quasi-familiarity of the process gave the team a sense of purpose and went some way to dispel hostility in a high pressure environment.

The Solution

We started by scheduling a weekly session and sticking to it. Sounds simple, but when the world is coming down around your ears, it is too easy to get caught up in all the reactivity and not make the space to think through what you're doing.

We set aside some time at the end of each week, and our format for the session was fairly simple:
• All the members of the delivery team state their level of confidence that the project will hit its deadline by calling out a number between 1 and 10, 1 being definitely not, 10 being definitely so.
• We record the numbers, then give the lowest and highest numbers an opportunity to speak, uncovering what they know that makes them so much more or less confident than the rest of us. This way the whole group learned about (or dispelled) issues and opportunities they may have been unaware of.
• In light of the new information from the outliers, everyone has an opportunity to revise their confidence estimate. This is recorded in a spreadsheet which lends this post its title.
• Finally we took some time to talk over the most painful issues or obvious opportunities in the project and the team, then picked the most critical one and committed to changing it over the coming week. We also reviewed what we promised ourselves we'd change last week.

Through this discipline the team made, in real time, a bunch of very positive changes to themselves and how they work together without having to stop working and reorganize. We also had a very important index that we could use to gauge exactly how worried we should be - which is pretty important from a risk mitigation perspective. The trend was important too - we were able to observe confidence rise over time, and respond rapidly to the dips which indicated that the team had encountered something that they were worried would prevent them from delivering.

The Outcome

This exercise gained us a much more stable view of a chaotic process, and let us start picking off improvements as we worked. By making the time to do this and being transparent with the discussions and decisions, the team felt more confident and in control of their work - which always helps morale.

Because we were able to give the business a coherent, accurate assessment of where we were at, we were able to confidently ask for the right support - it was easy to show where the rest of the organization could help, and demonstrate the impact if they didn't meet their obligations to us.

In summary, we got our issues out in the open and got our project back on the rails. And by the time we got together for our post-implimentation retrospective, we were pleasantly surprised by how many of our most critical problems we'd already identified and fixed. If you're in a tough spot with a significant piece of work and a fixed deadline, consider giving something like this a try - I think it will work alongside any development methodology.

My agile construction yard

I have been successfully practicing agile for quite a while now, and I've always believed that, given a pragmatic application of the principles behind it, it can be used to manage any process. Mind you, having only ever tried to deliver software with agile, this remained personally unproven. Gauntlet?

So I figured that if I am going to keep going on about it, I am going to have to put my money (and my house) where my mouth is. So I did, and this is the story...

The requirements

I wanted a bunch of work done on my house - extending a room, replacing the whole fence line, building a new retaining wall, laying a stonework patio, new roof drainage, building a new BBQ area, and some interior layout changes - and I thought it's now or never. I spoke to the builder and told him all about agile, lean thinking, project management practices like SCRUM and XP, and how it can benefit both of us in delivering House 2.0. He asked if he could speak to the responsible adult who looks after me. "Great, a waterfall builder" I say to myself as I try not to be offended by the 'responsible adult' quip.

But we strike a deal; he's going to do big up front drawings and a quote, and we'll proceed my way at my own risk and responsibility. The game is on.

The first thing we do is run through all the things I want done, which ones are most important to me, and roughly how I want them all to look. I guess you could call this the vision setting. Then my contractor asks me the few big questions; the materials I want to use, budget, and when I want it ready by. He makes some high level estimations on time and cost, based on which I rearrange my priorities to get a few quick wins. We have a backlog.

The project

Our first 'sprint' is the fence line. We come across our first unknown already - the uprights that hold the fence into the ground are concreted in and we either have to take twice as long to tear them out, or build the new fence on the old posts. Direct contact throughout the process and transparency of information ensures that we make the decision together, as customer and delivery team, so that neither of us is left with the consequences of unforeseen situations. I want the benefit of new posts so I'm quite happy to eat the costs put forward.

Next we do the retaining wall, and we have a quick standup to go over the details - we need to decide on a few details up exact height and the type of plants growing across the top. Since the fence has been done I go with some sandy bricks that match the uprights and the wall is constructed without incident. The next thing we're going to tackle is the BBQ area; however, beyond that the roadmap calls for the room extension and so we need to apply for planning consent in order to get the approval in time. Agile doesn't mean no paperwork and no planning, it means doing just enough just in time for when you need it.

Now we hit our first dependency - the patio must be laid first before we can build the BBQ area. That's cool, and through our brief daily catchups, we come up with an ideal layout and pick out some nifty blocks. A bit of bad weather slows down the cementing phase slightly, but we're expecting that - this is England. We use the opportunity to draft some drawings for the room extension and get the consent application lodged.

It's BBQ building time. I've been thinking about it since we started the project, and I decided I wanted to change it. The grill was originally going up against one wall, but wouldn't it be much more fun if it was right in the middle so everyone could stand 360 around it and grill their own meat? You bet it would. We built a couple of examples out of loose bricks (prototypes?) and then settled on a final design. It takes a bit more stone than the original idea, but it's way more awesome.

Then our project suffered it's first major setback - the planning consent process uncovers that a whole lot of structural reinforcement will be needed if they're going to approve the extension. That pretty much triples the cost of adding the extra space. Is it still worth it at triple price? Not to me. Lucky we didn't invest in a lot of architect's drawings and interior design ideas, they'd be wasted now (specifications are inventory and inventory is a liability). So we start talking about alternatives, and come up with a plan to create the new space as storage and wardrobes - not exactly what I had in mind up front, but at less than half the original cost it still delivers 'business value'.

The retrospective

So how did it all turn out? Well, as a customer, I felt involved and informed throughout the whole process, and the immediate future was usually quite predictable. Throughout the project I had the opportunity to adjust and refine my requirements as I saw the work progress, and I always made informed tradeoffs whenever issues arose. I am happy, the builder is happy, and I got exactly what I wanted - even though what I really wanted ended up quite different to what I thought I wanted when we started.

Oh and if anyone wants a good building contractor in Surrey...

And the problem with agile is...

I get drawn into a lot of debates about the pros and cons of agile (or maybe I just like a good fight?) and the standard attack pattern around quality is starting to become sooooo passe. So I'm going to tackle it here, in the hope that I can use a one-link defense in future. Kind of like TinyURL but for arguments.

Firstly, let me go on record by saying that I most certainly am an agile proponent, but at the same time I don't believe it is the single, solitary answer to how we should do everything in technology. You have to have more than one tool in your kit.

So let me go ahead and get this off my chest:

Agile != doing a poor job

Agile doesn't mean doing no testing, it means doing just enough testing to ensure appropriate quality.

Agile doesn't mean doing no documentation, it means doing just enough documentation to effectively convey meaning and understanding.

Agile doesn't mean doing no architecture, it means doing just enough design work to understand how to start iterating the solution.

Agile doesn't mean having no requirements, it means having just enough detail for the foreseeable future and embracing change beyond that.

It is also about appreciating that just enough can mean different things in different projects, different business problems, and different parts of the system.

Come on people, these aren't new ideas!

To be fair, I can see where some of this criticism comes from. There are cowboys out there who use agile as an excuse to dispense with necessary diligence or take ill-advised shortcuts. When it all comes crashing down, it does sound better to say "hey, we were doing agile" than "hey, we couldn't really be bothered planing this work properly" for the individuals concerned. The fact is that crappy engineers exist. There is no such thing as an SDLC that turns good engineers into poor engineers, we just started accepting that as an excuse.

If agile does have something to answer for here, it is that this kind of poor work is much more visible much earlier (if concealment is your game). The reality is the same team would make the same (or worse) errors in judgement regardless of the approach they used, you just wouldn't know about it for 12 months - and by then, who knows whose fault that was?

Don't accept output any crappier than you otherwise would just because it has agile stamped on it - if anything you should expect better, because you will have had more opportunities for course correction along the way.

Agile Architecture

There are a lot of opinions on how to handle architecture in an agile SDLC - so I thought why not throw mine out there too?

The first thing I think is important is to maintain a good reference architecture that readily explains the whole system including shared infrastructure and how to access it.  Having strictly defined interfaces/APIs and communications infrastructure rules is fundamental to being able to build anything with a distributed system as a platform.

Your biggest need for architecture is going to be early in a new project to build a new feature or product - so for the first few sprints reserve extra long times for sprint planning and use that time for consultative design sessions.  Lots of people use a "design sprint" or two but I've never been a big fan of that - I think a sprint is the best use of time once you've planned out exactly what tasks you're going to do and have an estimate (based on facts) of how long they're going to take.  The dedicated, concentrated burst of activity that traditionally makes up a sprint isn't the environment most conducive to the contemplative, consultative, learning process that is design.  Besides, having a designated "design sprint" tends to make you want to answer all the questions and design everything up front - and we all know that's kidding ourselves, in fact it's probably why most of us adopted agile in the first place!

Next you should make your first sprint (and perhaps even the second depending on the size of the project) long enough to build the entire width of the whole system to the barest, most skeletal framework possible (a month for example) and the depth one customer feature to [one normal sprint's worth of] completeness.  Then you can go back to your shorter, more focused, high energy sprints (2 weeks for example) and flesh out the rest of the system feature-by-feature.

This works on the theory that "deliver working, demonstrable software every sprint" overrules "many short, predictable cycles of delivery" and in any case I believe the output reflects the right mix of performing looser, more creative solution architecture activities and tightly timeboxed, ferocious task burning activities.

Get Attached to Results

It seems a little obvious to say but results matter - the reason we adopt certain processes or take certain actions is to create a desired effect, not just to exercise our procedure muscles.

All well and good but so often I run into people who are so wound up in the process, so attached to checking off the steps like a to-do list that they forget to look for the change they're supposed to be making by checking that list off.  This goes for everything from project wash-ups (looking to improve the SDLC) to performance reviews (looking improve the quality of individual's contribution and give them a workable career plan) to basic, everyday meetings (looking for a decision or consensus in a group).

Keep your eyes on the prize.  If you get to the end of a process and nothing is different you are not done.  Go back and try something else.  Get attached to creating the difference, not stepping through a script.

Is Agile Gaining Wait?

No that's not a typo, just my rapier-sharp wit (ahem).  Reflecting on us as an industry I am worried that we're doing to agile what we did to SOA - productise it.  SOA has become such a blurry, distorted concept that there are now legions of consultants making - what do the kids call it these days - fat cash just explaining it to companies.

How did we get there?  Well we started off with classical distributed computing then came up with a thought process (SOA) we could lay over the top to help design distributed architectures that better reflected the business processes they supported.  That made the concept a little more accessible and suddenly a lot of organisations wanted it on the action.  Cue vendors because hey, if someone wants something we can always find a way to license it.  I used to call SOA distributed systems with some conventions (rules and patterns) but now I call it distributed systems with some conventions (sales conferences).  There's that wit again.

But we're on a tangent - back to the point:

Are we doing the same thing to agile?  Are we diluting the power of its concepts by trying to shrink wrap it into something we can market, ship and pay maintenance on?  If you look at the original agile manifesto (arguably where it all began) you'll read some pretty agreeable statements of principle, like "individuals and interactions over processes and tools" and "responding to change over following a plan" etc.  I'm all for that but what scares me is every day I see more tools and processes 'for agile' and more consultancies teaching and preaching it.  I'm getting asked questions like "what's our process for agile?" and "this book told me to always do X" more frequently.

Don't get me wrong I believe tools are important and as for all the preaching - well if something is valuable and can help us as a profession then we should spread the word.  Being an evil capitalist at heart I also don't see anything wrong with getting paid for that, I'm just saying we need to keep our eyes open for the boundary between this and when we start to reduce it's effectiveness.

I guess what I'm saying is I just don't want to overcomplicate and burden with process something whose greatest strengths are its lightweight simplicity and use of old fashioned common sense.