The difference between taking a shortcut and cutting a corner

There are two very different things to talk about here, with two very different implications for your software, but I'll admit that which way around you label them can simply be language pedantry.  So for the purposes of this discourse, let's say that:

• Taking a shortcut is using your knowledge and experience in place of studying things from first principles every time.
• Cutting a corner is a compromise in quality or good practices in the place of the prudent actions otherwise required.

Both are used to speed progress through a given piece of work.  Using knowledge and experience is a time to market competitive advantage gained through having talented people, compromising quality is a time to market advantage gained through process; albeit purchased at the price of later cost.

Taking Shortcuts

Let's talk about taking shortcuts first.  When you can do this confidently it is a magical place to be.  I like to hire really smart people and give them the freedom to use the experience they've gained throughout their careers to leap frog us straight over the other organisations who had to do it all from scratch, the sources of the experience I want to tap into.

Experienced people have been through years of doing everything by careful analysis and research, gradually proving out a larger and larger set of assumptions which they can build up from, and using life's great feedback loops to tune their thinking until their instinctive decisions are as a good as a noobie's 3 months of science.  That's why we hire the kind of people we hire.

Having all that collective wisdom at hand and not using it is irresponsible leadership and suffocating to the innovators you should be growing.  There are still times when the right thing to do is take things back to first principles and do that 3 months of science, but the beauty of experience is it also tells you when it's right to use your instincts and when it's right to figure it out.

Cutting Corners

And now cutting corners.  Reducing time to market but doing something less well (sacrificing quality) or by skipping validation steps such as doing less testing (taking more risk) works in the sort term, but really just exchanges a short term temporal win for a lot more trouble later.  Often more later trouble than you really think, which is why this accelerent is so frequently misused.

I say 'freqently misused' quite deliberately, because in a pragmatic real world it is not always the wrong thing to do.  Occasionally it can even be the path to very big wins, when time is absolutely of the essence and every day counts materially.  But it is always the wrong thing to do to unconsciously blunder into cutting corners, to not be actively managing your quality, and to incur this type of technical debt without having a credible plan to come back to fix it up before it festers too long.

If you are not grown up enough to manage the payback, then you are not grown up enough to incur the debt.

So...

Hire people smarter than you are.  Empower them to use what they know and what they've done for your benefit - that's what they want anyway - and to use their time with you to further expand their horizons.  Carefully manage your quality, incur technical debt strategically, and never do it without paying it back.

You'll do a good job and everyone will have a good time, most importantly your consumers.

Software vs. Infrastructure

For some reason I’ve been running into a lot of other CIOs and CTOs lately and one topic that keeps coming up was whether we were development or infrastructure focused. The whole time I kept thinking to myself why is this even a question? Why do we feel like we should choose? Or, somewhat more provocatively, why do so many of us only feel like doing half the job?

I concede that there is a ‘background’ element to this. No one I know started off as an IT executive; we were all DBAs or developers or sysadmins first, and that gives us a nice comfortable area of personal expertise we can use as shortcuts but it shouldn’t set our agenda in a leadership role which [in most organisations] encompasses the whole shooting match.

One of the reasons why many organisations suffer the good old fashioned laundry list of regular technical woes (that we could probably all reel off by heart) is because of the interplay between development and infrastructure and a lack of end-to-end oversight over both. And, as the technology leaders, if we don’t understand both sides, keep strongly engaged with them, and create teamwork and cooperation where there has typically been borders and a lack of mutual interest, then who will? There are very few other roles with a remit in both areas and even fewer that should be taking responsibility for them failing to be an end-to-end unit.

There is a whole ‘devops’ kind of meme swelling up these days – and more power to it; I think it is dead on the money.

Devops is, by nature of the organic meme it’s emerging as, poorly defined but to me it is about the recognition that [in most cases] customers care about product – not software or systems – and product is that butter-smooth combination of software and infrastructure finely tuned to work together and operational know-how to keep it running.

It’s also about having the right feedback loops in place between engineers and the real world, so that as your products get used and abused, they become more fit for that very purpose.

Technical Debt Credit Rating

A while back I wrote about how closely technical debt paralleled financial debt and then a short follow-up post with the meta-message that, through the right reductionism, certain things will always cost you an inescapable minimum and the decision that you really have is about where you choose to make that payment (I argued that you either pay through the correct design and build initially, or you pay through unnecessarily increased maintenance costs later).

Outlawing technical debt is as foolhardy as always cutting corners - sometimes the right thing to do is to take a hit to get something out sooner or in a certain shape - the problem is in the repayments; in other words, they often don't exist. And if you don't go back and sort your hacks out early they tend to accumulate in the system and before you know it developing on it is like moving through treacle (and the treacle occasionally catches fire). Everything else you do takes that much longer and is that much more expensive (and those are business metrics) and can have unintended consequences - it's a crippling burden just like its financial counterpart.

I have a rule - no one is allowed to take technical shortcuts through a project [where that = incur some technical debt] without showing where, in the same plan, they go back to do some refactoring so that our long-term assets remain sustainable. We're pretty lucky at Sporting Index; most of our stakeholders value technology and recognize how critical it is to the business, so it's a conversation I rarely have to have - but I know many other companies are less fortunate.

So now I think I have a better idea. It came to me while I was thinking about why, if technical debt is so similar to real debt, it doesn't regulate itself in the same way - and I think that's becuase there is no concept of creditworthiness in technical debt, in other words no assessment of a team's ability to 'repay' a quality compromise (read: loan of functionality to the business ahead of when it otherwise would have been ready) alongside all the other outstanding quality compromises that they already have.

But we might be able to simulate that...

Imagine giving each product manager a 'credit rating' of, say, 3 - meaning they could call for unsustainable technical shortcuts 3 times as they worked with the delivery team to determine their solutions. Once those 3 debt points were used, that product manager would be unable to encourage the team to make further quality compromises until at least one of those was paid back. If you have product managers with a demonstrated track record of revisiting earlier deliveries to address technical debt then why not give them 5? A higher creditworthiness ought to enable an individual to take on more concurrent debt as we can be more confident that they'll come back to service it.

I pretty much guranttee that any product manager's cavalier attitude towards shortcuts would change and they'd suddenly treat technical debt as the instrument it really is and use it only where it mattered materially. That, ladies and gentlemen, is a feedback loop.

The downside is that you'd have to be better at tracking this sort of thing and, like any other method, it'll need some discipline when a product manager with maxed out technical debt points wants just one more shortcut. If you've ever had to maintain a spaghetti codebase (as most of us have at some point) then you'll know that these things need closely policed anyway, so perhaps being forced into formally tracking it isn't so bad.

Just a little more on SLAs…

Yesterday I posted a little something about SLAs and I’m always happier with things when I can wrap them up with a handful of guidelines.  Not always possible in the complicated world we live in, but here goes anyway:

1. Discover the things that are meaningful for the business.  I risk stating the obvious but there is always a temptation to approach this ‘backwards’ by starting off with what can be measured rather than what is significant (and then working out how to measure it).  You don’t want to end up with a bunch of metrics that are easy to count but don’t describe desired system performance.
2. Strike a balance between persistence and change.  Unless doing lots of projects isn’t important to you, be careful not to base all your KPIs on availability/stability metrics – or if you do, at least be aware of how that can drive reluctance to push changes through the system.
3. Make appropriate interpretations for each product or system.  In most organisations different systems, or parts of each system, are subject to different uptime, capacity, latency etc demands.  And assuming you pick some basics like performance they should be specific to each product; for a website that might be a number of page impressions, and for an analytic system that might be a time to render when a data set is updated.
4. Include time as a dimension.  Most businesses – particularly on the web – have a number of 24x7 products, but there are also a lot of systems that only get used during business hours or at certain intervals (e.g. payroll is usually a monthly thing).
5. Disregard #1.  Kind of.  Now that you’ve gotten this far, you will need to consider some feasibility, because signing up to unachievable SLAs doesn’t help anyone.  Have a look at what devices and services underpin the business functionality you are measuring.  Trees of dependencies, composites in SOA for example, tend to live up to the least strict SLA rather than the aggregate of the set.

Rules of thumb – apply in conjunction with local knowledge!

More Meaningful SLAs

Establishing internal service levels with the rest of the business is a difficult process - there are so many variables that can be measured and, as we all know, you change what you measure by measuring it. For example, if you express your SLA exclusively in terms of system uptime, then you improve all the activities around keeping your system available. The flipside of this is that you often discourage the activities around effecting change in the system - after all, any releases or upgrades or new features always carry some risk to availability and that's what they're measured on...

The place to start is to work out what's important to the organisation. Performance and availability are critical to us (a latency sensitive transactional platform with variable usage patterns) but so is change (a content driven web application correlated with events in the real world). We decided that performance, availability, change, and support response were the key metrics for us - nothing unique so far, and next we had to make an interpretation of each of these that was relevant to our various systems.

A basic principle here is recognising that it isn't just the raw numbers that should be appropriate to each individual product, but what is being measured too. Throwing an overall value at the problem (for example 99% availability across the board) makes the job of putting together your SLA easier, but is it a true reflection of your infrastructure? Whenever I've seen this coarse-grained approach used it has always led to less than acceptable uptime for the most critical applications and wasted investment propping up others that are realistically less important.

Another way to make sure your SLAs really closely matches business need is to introduce the dimension of time. In many systems and many organisations demand - and the cost of downtime - varies over time. For example, how many accounts and payroll systems are used around the clock? If you can trade off to 'best endeavors' over weekends and evenings then you shouldn't have too much trouble meeting a five nines commitment during business hours between Monday and Friday.

For our website we have a flat availability target (such is the nature of a 24x7 site) and performance we interpreted in a latency metric for price publishing and order placement. For reporting systems - which do not experience the same round-the-clock demands - we have different availability targets during business and after hours. Performance in the context of those systems is interpreted as a certain set of daily reports delivered by a fixed time each morning and a message delivery SLA on alerts on certain events. SLA's around change and product delivery are much more complicated and fraught with subjective measures. We've gone with measuring development projects iteration-by-iteration; what got delivered vs. what was committed during that sprint's planning. It's objective and encourages good estimation and strict control of scope creep during a sprint.

Making SLAs commensurate with what the business genuinely demands from a given piece of technology is important. Setting your sights high can seem like a good idea on the surface but, when you consider the frightening magnitude of difference in cost between 99.5% and 99.9% uptime, that couple of points can only ever be described as waste if they are not intimately linked to the organisations success.

Gates and Nonblocking Delivery Practices

A few weeks ago one of my guys sent me this article from The Agile Executive which is - although not explicitly said - about the collision between ITIL infrastructure governance and agile development methodologies. I use the word 'collision' very deliberately because those guys deserve credit for how well they've managed to knit those (often opposing) worlds together. Which brings me nicely onto gates.

Gates, defined as the "You Shall Not Pass" checkpoints which must be navigated throughout a project, are not inherently bad. They just get misused in the same way that anything we do has the potential to be applied over zealously.


Gates have a place in delivery and fulfill an important function. They serve as a kind of final quality checklist to make sure that all those things you said mattered and had to be done actually have been done. As long as they are few, important, and near the end of the development cycle, you can make this work.

Where this tends to go wrong is when gates are established without clear and measurable clearance criteria defined up front. What you have then is an obstacle with a variable and subjective success criteria - and then everyone wonders why a project arrives at the gate and struggles to meet the requirements to move on. An understanding of exactly what's on the checklists a project will face as it goes live and exactly how that will be measured has to be front loaded, so that development teams know exactly how to make a piece of work transition through the lifecycle smoothly and organize resources in advance.

If this isn't established early and you get a project logjam downstream then you often end up having to compromise - and that means compromising on key quality or operational requirements that you believed are important enough to gate - in order to restart the flow. The good old fashioned 'defects fixed later cost more' curve still applies here.

Another common misuse of gates is introducing them into a process primarily for reporting purposes. It is true that a regular set of gates established throughout the end-to-end project process provides a set of convenient handholds against which to benchmark progress (we are at 'customer feedback on specification' or entering 'test run 3' etc) but there are a couple of downsides that go along with that.

Firstly, it tends to encourage over-reliance on artifacts such as documentation and reports rather than working software as a primary work output. Secondly, progress becomes measured by the clearance of gate after gate, step after step, and revisiting a previous step is seen as a step backwards. That lines up with an easy to describe liner view of the world, however with most complex projects in most organizations things are not that simple and progress often involves degrees of overlap and leapfrogging.

Right First Time

Imagine if the first wall you ever built had to form an integral part of your dream home, or if the first time you ever drew a picture it had to hang in the Louvre? Or if the first cake you ever baked had to be for your daughters wedding?

That’s clearly unreasonable.

So why does it seem reasonable that the first iteration of a new system you build should be the one that your business depends upon?

That’s why we prototype. That’s why we iterate. That’s why we beta. Don’t rob yourself, and your organisation, of the wisdom you’re owed from those lessons.

Don't out-specify your knowledge

Combining some points from this post by Joe Armstrong and this post by Dan Creswell gives rise to some interesting implementation thoughts.

Standardisation is important, particularly in the context that Armstrong frames it in - if the interaction between a client and a server closely adheres to a predetermined protocol, then the systems overall behaviour becomes more predictable and easier to adapt and fault find.

But then this depends on the maturity of your understanding of your problem and your solution. As Creswell says at the beginning of his post, defining too much up front can be a trap. Coming up with a set of standards that are too rigid too early might start you down a road of building that system instead of building the right system.

Standards, patterns, and contracts are all important - but timing is key.

You ALWAYS Pay for Quality

What if I told you that every single organization pays the same for any given piece of functionality? You'd probably call me crazy, and that would probably be because you know that it's not true, right? You've negotiated different rates, done projects in different locations, used different vendors, and cut different corners - and consequently, it cost you a different amount of cash each time. So I can't possibly be right, can I?

But before you condemn my point entirely, consider this:

If you're buying in software, consultancy, per-project services etc, then the primary levers you can adjust are commercial terms. You might feel exceptionally good about reducing the cost of a project by £5K or shaving a couple of points off margin, but do you really think those suppliers are going to produce exactly the same output for you and just make £5K less on the deal? Good luck with that.

If you're building your own products, and assuming you've got a fixed size team, then the primary levers you can adjust are tradeoffs and shortcuts - i.e. where you spend your time. How you pay for quality is then divided up between how much you're prepared to pay for now up front, and how much you're going to pay later - typically through higher operational or technical support effort (less automation), increased cost of the next features (technical debt), and opportunity cost (revenue lost to product unavailability).

So I'd suggest that you always pay for quality, whether you intend to or not, and I'd add that it's usually less painful to pay for it overtly through a proper project plan, than inadvertently through reactive support, interest payments from future projects, and your customers patience.

I'm not saying that you should never try to sharpen a deal to save some budget or take a shortcut to get a product out quicker, simply that you should keep in mind the longer term costs as you congratulate yourself on the shorter term gains.

No Pain No Gain?

For the past couple of days I've been helping out a friend get a new team organized, and during our of discussions in their canteen, I overheard something that epitomizes the complexity-aholics that really drive me mental.

A couple of data architects were talking about a consolidation project they were kicking off. "Can you believe what they're doing in the Japanese office? Their schema is so basic" one complained to the other. "Yeah, they clearly don't know what they're doing" his colleague replied, "they should leave it up to us." A couple more minutes of the same and I couldn't really let it go.

Deep breath...

Gathering up my most diplomatic tone, I asked the resident geniuses: "OK lads, if you're saying that they have a basic, simple solution which meets the business requirements, and you have a complex, difficult solution which meets the business requirements - then isn't it you who have something to learn from them?"

You see this is why I don't have any friends.

But seriously, what's the attraction to building big, creaky, behemoth solutions where something easy and elegant would meet everybody's expectations? Why is it that we seem to derive satisfaction from making our lives hard? Unless you're very lucky (lazy?), you'll have to troubleshoot, maintain, and support whatever you put in place.

I've noticed before that good engineers can build vast systems with webs of dependencies and many moving parts, but it's the great engineers who realize that this is a bad idea.

Outside of the gym, there are very few places in the world where you'll be better rewarded for increasing the effort necessary to achieve a given outcome.

The Parable of the Big Hands

This is an old story, retrieved from the dusty recesses of my memory, but it's as relevant today as it ever was. Names changed to protect the incompetent...

A number of years ago I was called in to help an integration company who had been struggling for a number of weeks with a manufacturing client. They had delivered a factory automation system, introducing computer-controlled fabrication and handling machines, replacing the manually operated mechanical kind. The project had been justified upon increased productivity and reduced material wastage - yet reduced productivity and increased metal wastage were being observed. Not really the results anyone was hoping for, customers blaming the vendor, vendor blaming the users etc.

My job was to "troubleshoot the new software" and "find the bugs" before the manufacturing company started firing live ammo.

Step 1 was to get a coherent problem statement. It seemed that in the vendor's lab everything went according to plan - sheet metal machines could be reconfigured in tens of seconds via a new software load - yet at the factory it was taking minutes and materials were frequently being cut to the wrong dimensions.

So what was different between the factory and the lab? I was assured that there was no differences at all - machines had been swapped, the exact same builds had been uploaded to the machines at both sites, error logs had been downloaded etc. But clearly there was a difference, otherwise it would be working (or broken) in both places.

I figured there must be something environmental - how else could the exact same machine, software, and configuration work perfectly in one location and not another. "So, tell me about this factory" I asked. "Never been there" was the answer (believe it or not). Ah-ha!

So off we go to the warehouse/factory thingy. We never even got through the tour of the floor before we made the critical observation - these metalworking dudes have really big hands! If your job consists of throwing around huge blocks of steel, cranking big heavy leavers, and turning giant bolts, then you soon develop a pair of mitts like shovels.

Why was this important? Quite simple. The keys on the small numpad-type keyboard that had been specially made for entering the program parameters on each machine were a standard keyboard size, and were therefore simply far too small for the bunch-of-bannans fingers of the guys that had to use it. These guys are hardcore welders and whatnot, they don't have our delicate little programmer hands!

So what's the lesson here? Know Your Users. It's as simple as that. You have no business even touching a keyboard (of any size) until you know who your users are, where they work, how they'll interact with the system, and what successful use is. Remember - you're not building the software for yourself or your business analysts, you're building the software for them. Get to know them.

When is the right time to launch?

Ignoring commercial concerns such as marketing campaigns, complimentary events, and market conditions, this essentially comes down to weighing the quality/completeness of the product against getting to market sooner.

The quality over time to market side has been advocated a few times in recent episodes of Paul Boag's podcast, so I figured I would speak up for the other side. But before I do, let me just say that I don't think there is a right answer to this and, as Paul also conceded, it depends on if your application's success requires a land grab or not.

I am, by nature, an "early and often" man - and that's kind of a vote for time to market over quality. I say kind of because I think it would be more accurate to say that it's a vote for time to market over perfection.

For me, the "often" part is inextricable linked to the "early" part. If you can show a pattern of frequent, regular improvement and feature releases, then you can afford to ship with less on day one. Users can often be more forgiving when see things turned around quickly, and new things regularly appearing can even become a reason in itself to return to the site more often.

Quality is still a factor, but it isn't as black and white as I've been hearing it presented. In the early days of a new product, I think where you spend your time is more important than overall quality. You should be very confident in anything that deals with accounts, real money, or the custody of user's data before shipping. I would argue that getting those areas right for launch at the expense of other parts of the system is better than a more even systemwide standard of quality. You always have limited resources. Spend the most where it counts the most.

And finally, openness. Be truthful and transparent with your users. Start a blog about development progress and the issues you've run into. Provide feedback mechanisms, and actually get back to people who've taken the time to share their thoughts - with something material too, not just an automated thankyou. Send out proactive notifications ahead of impactful changes and after unplanned events. Stick 'beta' labels on things you're shipping early - it'll keep user's blood pressure down, and you might be surprised by how much of the community is prepared to help.

I am aware that I haven't actually answered the question that lends this post its title. I don't know if there even is an off-the-shelf answer, but I hope that I've at least given you some more ideas on how to make the right decision for yourself.

Failure Modes

Whenever we're building a product, we've got to keep in mind what might go wrong, rather than just catering mindlessly to the functional spec. That's because specifications are largely written for a parallel universe where everything goes as planned and nothing ever breaks; while we must write software that runs right here in this universe, with all it's unpredictability, unintended consequences, and poorly behaved users.

Oh and as I've said in the past, if you have a business owner that actually specs for failure modes, kiss them passionately now and never let them go. But for the rest of us, maybe it would help us keep failure modes in the forefront of our minds as we worked if we came up with some simplified categories to keep track of. How about internal, external, and human?

Internal
I'm not going to say too much about internal failure modes, because they are both the most commonly considered types and they have the most existing solutions out there.

You could sum up internal failures by imagining your code operating autonomously in a closed environment. What might go wrong? You are essentially catering for quality here, and we have all sorts of test environments and unit tests to combat defects we might accidentally introduce through our own artifacts.

External
The key difference between external and internal failure modes is precisely what I said above - you are imagining that your code is operating in perfect isolation. If you are reading this, then I sincerely hope you rolled your eyes at that thought.

Let's assume that integration is part of internal, and we only start talking external forces when our product is out there running online. What might go wrong?

Occasionally I meet teams that are pretty good detecting and reacting to external failures and it pleases me greatly. Let's consider some examples; what if an external price list that your system refers to goes down? How about if a service intended to validate addresses becomes a black hole? What if you lose your entire internet connection?

Those examples are all about blackouts - total and obvious removal of service - so things are conspicuous by their absence. For bonus points, how are you at spotting brownouts? That's when things are 'up' but still broken in a very critical way, and the results can sometimes cost you far more than a blackout, as they can go undetected for a while...

Easy example - you subscribe to a feed for up-to-the-minute foreign exchange rates. For performance reasons, you probably store the most recent values for each currency you use in a cache or database, and read it from there per transaction. What happens if you stop receiving the feed? You could keep transacting for a very long time before you notice, and you will have either disadvantaged yourself or your customers by using out of date rates - neither of which is desirable.

Perhaps the feed didn't even stop. Perhaps the schema changed, in which case you'd still see a regular drop of data if you were monitoring the consuming interface, but you'd have unusable data - or worse - be inserting the wrong values against each currency.

Human
Human failure modes are the least catered for in our profession, regardless of the fact they're just as inevitable and just as expensive. You could argue that 'human' is just another type of external failure, but I consider it fundamentally different due to one simple word - "oops".

To err is human and all that junk. We do stuff like set parameters incorrectly, turn off the wrong server, pull out the wrong disk, plug in the wrong cable, ignore system requirements etc - all with the best of intentions.

So what would happen if, say, a live application server is misconfigured to use a development database and then you unknowingly unleash real users upon it? You could spend a very long time troubleshooting it, or worse still it might actually work - and thinking about brownouts - how long will it be before you noticed? For users who'd attached to that node, where will all their changes be, and how will you merge that back into the 'real' live data?

Humans can also accidentally not do things which have consequences for our system too. Consider our feed example - perhaps we just forgot to renew the subscription, and so we're getting stale or no data even though the system has done everything it was designed to do. Hang on, who was in charge of updating those SSL certificates?

Perhaps we don't think about maintenance mistakes up front because whenever we build something, we always picture ourselves performing the operational tasks. And to us, the steps are obvious and we're performing them, in a simplified world in our heads, without any other distractions competing for our attention. Again - not real life.

And so...
All of these things can be monitored, tested for, and caught. In our forex example, you might check the age of the data every time you read the exchange rate value in preparation for a transaction, and fail it if it exceeds a certain threshold (or just watch the age in a separate process).

In our live server with test data example, you might mandate that systems and data sources state what mode they're in (test, live, demo, etc) in their connection string - better yet generate an alert if there is a state mismatch in the stack (or segment your network so communication is not possible).

The question isn't are there solutions; the question is how far is far enough?

As long as you think about failure modes in whatever way works for you, and make a pragmatic judgement on each risk using likelihood and impact to determine how many monitors and fail-safes it's worth building in, then you'll have done your job significantly better than the vast majority of engineers out there - and your customers will thank you for it with their business.

Change Control

With another December rolling around already, we head into that risky territory that we must navigate once a year - seasonal trading for many companies is picking up, yet now is the time when most support staff are trying as hard as possible to be on holiday. A tricky predicament - and what better time to talk about change control?

A lot of people - particularly fellow agilists - regard change control as a pointless, work-creationist, bureaucratic impediment to doing actual work. If it's irresponsibly applied, then I'd have to agree with them, but there are ways to implement change control that will add value to what you do without progress grinding to a halt amid kilometers of red tape.

Firstly, let's talk about why we'd bother in the first place. What's in it for us, and what's in it for the organization, to have some form of change control in place? Talking about it from this perspective (i.e. what we want to get out of it) means that whatever you do for change control is much more likely to deliver the benefits - because you have a goal in mind.

Here's what I look for in a change control process:
• The discipline of documenting a plan, even in rough steps, forces people to think through what they're doing and can uncover gotchas before they bite.
• Making the proposed change visible to other teams exposes any dependencies and technical/resource conflicts with parallel work.
• Making the proposed changes visible to the business makes sure the true impact to customers is taken into consideration and appropriate communication planned.
• Keeping simple records (such as plan vs actual steps taken) can contribute significantly to knowledge bases about the system and how to own it.
• Capturing basic information about the proposed change and circulating it to stakeholders makes sure balanced risk assessments are made when we need to decide when and how to implement something, and how much to spend on mitigations.

Ultimately, this all adds up to confidence in the activities the team are undertaking, and over time, will lead to less late nights and less reactive work.

And here are my rules of thumb for how change control should be implemented:
• Never let any process get in the way of doing the bloody obvious. If someone's on fire, you don't go and get the first aid manual and look up 'F' for fire.
• Change control can be granular, with stricter controls on more critical elements (like settlements), and a more flexible approach on lower impact or easier to restore elements (like content and feeds).
• Don't just take a off the shelf or copy another organization verbatim - this is the kind of thing that got change control the reputation it has - think about what you need and do something appropriate.
• Start small and grow up - it's easy to add more diligence where it proves necessary, but much more difficult to relax controls on areas where progress is pointlessly restricted.

So what do you actually do? As I said above, start off lightweight and cheap - a spreadsheet should do it, there isn't always the need for a huge workflow management database. Make a simple template and make sure you circulate it the way information is best disseminated in your organization (email, intranet, pinned on the wall - whatever gets it seen). Borrow ideas from your industry peers, but keep in mind the outcomes that best serve your circumstances. Most of all, identify the right stakeholders for each area of the system, appreciate the different requirements the applications under your stewardship have, and get into the habit of weighting risk and thinking before you act.

Here's to peace of mind - let's spend December at christmas parties, not postmortems!

Are you getting the most out of Quality?

Testing != Quality.

Now that we've gotten that out the way, let's talk about why.  Testing is an essential activity in any quality system, but it should never be the focus.  Testing should be used to verify that your quality is working, not to verify that your product is working - because if you have a quality process fueled by quality input then you can predict a certain level of quality product.

There is something to be learnt from looking at the history of quality, long before we needed to apply it to software.  Before the industrial revolution which saw mass manufacturing on production lines, all we had was testing.  Quality of output was a function of the individual craftsman - pieces were all made and inspected manually.  Clearly this process couldn't scale to outputs in the tens of thousands of units per day, as some production lines were capable of, and so we needed a way to ensure the quality of our output without having to individually inspect every single unit.  And thus, quality was born.

The key was focusing on the process; ensuring that materials, practices, and tools adhered to a certain standard to create consistent output.  This way, inspecting the output became the way we tested the process, not the units, and we could afford to take samples instead of reviewing every individual item.

I am surprised by how many organisations still have a 'testing the output' approach to quality, they could be spending their time so much more wisely.  There are a whole bunch of things that make up quality in software projects; the architecture, the specifications, the documentation, the environments, the project management, and the testing which proves it was all done adequately.

Invest in quality, not in testing, inspect samples only, and automate a complete exercise of all your functionality (what it should do and what it should not do) before you ship.

Planning

I like Dan's analogy on this, simply motoring along blind will almost always have a fiery end.  I want to apply it to planning on a more practical, daily basis though - I know this isn't exactly what Dan meant but I think the concept scales down with equal validity.

We know that whenever we break projects down into good quality user stories, then into individually-ownable tasks, collaboratively estimate these and then assign and track them in daily stand-ups we get consistently good results and predictable delivery.

The problem is that when the heat is on - for example one of Dan's crazy deadlines looms ominously - proper planning seems to be the first thing cast overboard to save the ship.  After all, whatever time you spent on planning is time you could be coding right?

We also know that whenever we try to tackle a huge volume of work in a very short time by just diving into it head first and all working like mad we get consistently poor results and low quality (and quite often we don't even get there at all).  We tend to leave the bigger, harder things to the end so chances are once we've failed we've left the most valuable bits behind.

Given that we know these 2 things why is it we keep doing this?

With proper planning not only do you have the highest chance of success in the first place you'll also be driving the car (rather than just switching the headlights on) so if you do run out of time you can take some consolation in the fact that whatever is left probably represents the least business value of everything you could have done for that project.

I am the veteran of many foolhardy deadlines with fixed scopes and I've observed that teams tackling this type of work in a controlled, ordered list tend to feel less stressed and make fewer mistakes.  It doesn't feel like the weight of the world is coming down on their shoulders and they're facing a huge, amorphous mountain of work that just can't be done...

Do the Right Thing

I got a few emails about this post and I guess it might have been a little too vague.  The conclusion I was trying to draw is that we need more than one tool in out toolkits.

As engineers it feels intuitively right to have a single process, a single way to tackle every activity.  Even though we say we don't we secretly quite like a good ole' fashioned IF/THEN.  IF code like this THEN test like that.

Sure, we always need a framework to work in, a reference process to fall back on - just don't get dogmatic.  Recognize that not everything you do needs to be treated exactly the same way (unless you have a really boring job) and stay flexible enough to be able to do the right thing in the right circumstances and take advantage of opportunity as it presents itself.

The best sort of engineering is about creating competitive advantage through technology - it's uncovering business value as you see it so whatever flavor your favorite bureaucracy comes in always make sure you're not structured to ignore opportunity.

A Revenue Share != an SLA

When white labeling etc with 3rd parties a revenue share as a commercial structure often seems attractive.  There are usually some obvious benefits (depending on your specific terms) such as a low entry cost to a new market/product and shared risk - perhaps even a painless exit strategy if it doesn't work out for you.

One mistake a lot of people make when negotiating these deals is being a little too lightweight on the service levels governing the part of the product supplied by the 3rd party.

On the surface it seems intuitive that with a revenue share agreement in place you can afford to be a little less draconian on any SLA - after all you have a shared interest in keeping the top line flowing right?  Possibly you can.  But first you need to take into account the differences between you and your partner.

A rev share usually is almost always top line based (as that's the most transparent thing you can both count) but what really matters to you is bottom line; the bottom line for you and your partner might be quite different.  If you are in a heavily regulated industry like online gambling then you'll have a whole lot of taxes and levies that need to come out of your share - ergo you're both operating at a different margin.  Difference number 1.

The other thing to remember is that a rev share motivates partners to do the most profitable thing - that's not always the same as the best thing for your business.  Let's say you have a bug on the 3rd party side (undocumented feature?) that you find particularly distressing.  It might be very costly for them to resolve and this, coupled with the difference in operating margin, means they just might not because it isn't the most commercially viable use of their resources.  Difference number 2.

I guess the summary here is just because you're "in it together" doesn't mean you should skimp on the SLAs - treat it like any other contractual agreement or you just might not get what you want.

Closing the Gap

Last week I posted about the gap between infrastructure and software.  I figured it might be a nice idea to talk about how we could address that gap instead of just having a whinge.

Thinking about why this gap exists in the first place it seems to me that most organizations are structured to create it.  They have this "developers in one team, infrastructure guys in another" kind of philosophy - organization by trade.  Then battle commences and they throw work back and forth over an imaginary brick wall between the groups.  Here is my code, no we can't support that it doesn't have monitoring feature X, OK it's back, no it doesn't work on this server, but it worked in the development environment... ad nauseam.

The next thing most companies try is fixing the problem with a hefty application of good ole' fashioned process.  This inevitably brings with it gates (signoff points in the development cycle) that engineers need to navigate.  Bob needs to sign off the monitoring capabilities, Tim needs to sign off the failure testing, Steve needs to sign off the compatibility with the live environment etc.  Congratulations, you've now built natural choke points (and if you're not careful key man dependencies) into your product delivery process - and we all know a machine can only move as fast as it's slowest part.

So what did you learn through your use of gates?  You learnt who needed to have a say in your product development, that's what.  This means the journey wasn't a complete waste of time - unless you stop here.  Now that you know who needs to work together to build your product in the best way possible why not team them up like that?

Organize ownership by product or feature or service - whatever the basic unit of your engineering output is.  Cross discipline teams containing all the skillsets necessary to design, build, test, deploy and support your system on a per service/feature basis will yield you the most appropriate solutions in the shortest possible time.  Embed all the necessary capability in every team - you'll need it every time anyway the only difference is the size of the communication gap.

I've suggested that product = software + infrastructure + operational know-how to run it.  That being the case why shouldn't a product team be made up of developers, testers, system administrators, architects and product owners?

A Good Quote

I talk a lot about failure in systems - it's inevitability and the things we can do to minimize it's impact and recover in a quick, automated fashion (note I never said "prevent" failure).

I recently came across a good quote on the topic that I think is really punchy:

"failure is not an option - it's bundled with your software"

I would credit it's author but I can't seem to work out where it came from - I've seen it tacked on the end of a few people's forum posts. I can't help but think they meant it a bit more cynically than I'm taking it but it's a good thinking point when you're delivering online services.