A brief lesson on company culture and local optimum

Culture governs what happens when the CEO isn't in the room - or something like that - is one of my favorite little soundbites.  Like all the most profound things I collect I'm not responsible for its genesis and, after a brief google search, I can't definitively pin down a source.  So I'm in the academically-uncomfortable zone of having to leave a quote uncredited... shiver...

I believe in principles and purpose guiding the actions of independent thinkers over prescribed activities and traversing an entirely predetermined course to an arbitrary long term goal (which discourages learning and responding to the environment).

The behavioural norms established in any collection of individuals will either enable or constrain that freedom.  That is why company culture is important.

At this point it is probably worth noting that I'm not making an argument for one leadership philosophy being inherently superior to another, I think there are many organizational pursuits which are well suited to the repeatable (ie low variation by design) and predictable activities yielded by closely micromanaging highly prescriptive processes.  I simply argue that culture happens, whether you actively choose it or not, and you always benefit from understanding your nature and actively creating a culture within which it will be the healthiest.  In the creative and scientific fields I usually act I have a formula that works; based on learning loops, some good old AMP, and not trying to tell people smarter than I am what to do.

 

Like most people, my first few executive roles were the first chance I had to run things however I wanted, to really test my values in the real world without compromise.  As a potentially unsafe generalisation, you pretty much get to set up your department/division/whatever however you see fit.

And so I did.  And I wrought most excellent, high performing teams, who were quick to adopt and enrich ideas and then turn those ideas into some remarkably successful products.  And we all loved every second of our time together, and we always left on Friday smarter than we arrived on Monday.

That was the outcome I'd hoped for, and I was glad to see some of the principles I held dear proven in the dispassionate and indifferent real world; where the strength of feeling you have for your ideas has zero influence on how effective they are.

But that wasn't the lesson:

I had created these cultures as microcosms within larger organizations which had dissimilar global cultures.  I was fooled for a long time - because it worked so well with only a little friction at the edges - but the real test is what happens when you're no longer there to perpetuate it.  To keep this kind of microcosm going within an incompatible host requires constant force to protect the values and establish/defend the space for creativity and (critically) failure.  You can do it, and you'll achieve what you want to achieve, but all systems normalize over time and without the constant force entropy kicks in fast.

One of the things leadership is about is creating lasting change, a journey that is bigger than yourself and continues with or without you, so if you're serious about culture and you want to make a sustainable difference, then you need to infect the host (or just be the CEO!).  You can use your microcosm to prove the effectiveness of a different set of behaviors but, if you can't cause that change to ripple outwards, then it will most certainly revert when you take yourself out of the situation.

And, no matter how new you are in a role and how difficult it is to imagine moving on, you will.  Unless you suck.  Great people always move onwards and upwards; it is so with your teams (and if you're a good boss then you will encourage and enable it) and it is so with yourself (and if you have a good boss he or she will encourage and enable it).

Think about what you'll leave behind, and how that will be perpetuated without your influence.

Why don't we invent more?

I sat in on a conference call today (actually I'm still on it!) where we talked a lot about innovation - specifically why we haven't done more of it.  Lots of different stories but the common theme seemed to be governance.

When I say governance I mean things like roadmaps, product councils, architectural oversight, PMO, etc.  Basically the constructs we've set up to steer our technology investments and spend our resources wisely.  Properly applied, they are critical to the success of any significant engineering endeavour.  Improperly applied they are critical to the failure of any significant engineering endeavour!  So are they being properly applied to invention and experimentation?

I don't think it is a case of proper application as much as it is a case of application at all.  I think any engineer has two jobs; serve the roadmap - build the applications and systems defined and managed by our governance processes - and serve the technology - discover new ways of solving our business problems, advance your own knowledge by experimenting, come up with new ideas and see if they fly.  If you're only doing the former then you're only doing half your job.

These are two very distinct types of activity and they shouldn't be governed by the same controls.  It just makes no sense to apply ROI and risk controls and rigid scheduling to a journey of discovery with a totally unknown destination.  The difference between teams who innovate and teams who just talk about it is recognising the immense value in the journey alone...

Besides, you don't need all that bureaucracy.  All you need to be great at this is to leave a space for it to grow and - if you have the right people - it will expand to fill it.  That is how I roll and that is what I want to see.

Hi, I’m Eachan and I’m a fail-a-holic

I attended a Harvey Nash CIO leadership forum a few days ago, subtitled The Strategic Insight Survey. Combining words like “strategic” and “insight” as a standalone sentence usually sets off my spider sense, however I must say it was a most worthwhile evening. Other heavily overused and frequently misunderstood words that were thrown about with reckless abandon were innovation, transformation, and ROI etc – all against the backdrop of this current economic climate.

All this talk of innovation, and how you measure its ROI, was starting to tickle my aforementioned spider sense, but since a couple of the panel were from the public sector, I couldn’t help but want to explore the effect that the general public’s reaction to failure has on the culture within those organisations. This is why I’m either never invited back to these things or I’m on the panel next time.

Firstly, let’s just agree that the cost of innovation is not in salaries paid to folk you can point to and say, “he’s doing some innovating” or in capital expense on new widgets and bleeding edge technology. The price of innovation is failure. In fact, I would even suggest that the price of learning anything material is failure.

So then I look at the slating that any public service gets when things go wrong. We bay for blood in the most public of forums, newspapers, radio, and even (opportunistically) on the opposition benches. A major mistake was made, so someone has to go, right? There are countless examples of this; ranging from embarrassing (eg HMRC data loss), to genuinely horrific (eg Baby P tragedy), to that thin line between the rules and the spirit of the rules (eg MP’s expenses scandal).

It’s difficult to come to an agreement on what should have happened in any one incident, as we will all feel differently about each incident and will be affected differently by it, but that’s OK, because it is the pattern that matters. The pattern is what sets the culture.

If, every time things go wrong, we excise every last individual involved, how then do we ever hope for the lesson to be learnt? Who has felt the pain, and is carrying that forward as a driver for change? Where is, to quote a buddy of mine, the feedback loop?

Is it right for us to do this, to create such epic risk aversion, and then still expect these organisations to deliver even the most basic of improvements let alone giant innovative leaps forward? Are we just robbing them of the best ingredients needed to create the huge changes we’re demanding?

When I think more about it, I wonder if I really am a fail-a-holic at all. Perhaps I only embrace failure so much because I’m actually a success-a-holic…

Failure Modes

Whenever we're building a product, we've got to keep in mind what might go wrong, rather than just catering mindlessly to the functional spec. That's because specifications are largely written for a parallel universe where everything goes as planned and nothing ever breaks; while we must write software that runs right here in this universe, with all it's unpredictability, unintended consequences, and poorly behaved users.

Oh and as I've said in the past, if you have a business owner that actually specs for failure modes, kiss them passionately now and never let them go. But for the rest of us, maybe it would help us keep failure modes in the forefront of our minds as we worked if we came up with some simplified categories to keep track of. How about internal, external, and human?

Internal
I'm not going to say too much about internal failure modes, because they are both the most commonly considered types and they have the most existing solutions out there.

You could sum up internal failures by imagining your code operating autonomously in a closed environment. What might go wrong? You are essentially catering for quality here, and we have all sorts of test environments and unit tests to combat defects we might accidentally introduce through our own artifacts.

External
The key difference between external and internal failure modes is precisely what I said above - you are imagining that your code is operating in perfect isolation. If you are reading this, then I sincerely hope you rolled your eyes at that thought.

Let's assume that integration is part of internal, and we only start talking external forces when our product is out there running online. What might go wrong?

Occasionally I meet teams that are pretty good detecting and reacting to external failures and it pleases me greatly. Let's consider some examples; what if an external price list that your system refers to goes down? How about if a service intended to validate addresses becomes a black hole? What if you lose your entire internet connection?

Those examples are all about blackouts - total and obvious removal of service - so things are conspicuous by their absence. For bonus points, how are you at spotting brownouts? That's when things are 'up' but still broken in a very critical way, and the results can sometimes cost you far more than a blackout, as they can go undetected for a while...

Easy example - you subscribe to a feed for up-to-the-minute foreign exchange rates. For performance reasons, you probably store the most recent values for each currency you use in a cache or database, and read it from there per transaction. What happens if you stop receiving the feed? You could keep transacting for a very long time before you notice, and you will have either disadvantaged yourself or your customers by using out of date rates - neither of which is desirable.

Perhaps the feed didn't even stop. Perhaps the schema changed, in which case you'd still see a regular drop of data if you were monitoring the consuming interface, but you'd have unusable data - or worse - be inserting the wrong values against each currency.

Human
Human failure modes are the least catered for in our profession, regardless of the fact they're just as inevitable and just as expensive. You could argue that 'human' is just another type of external failure, but I consider it fundamentally different due to one simple word - "oops".

To err is human and all that junk. We do stuff like set parameters incorrectly, turn off the wrong server, pull out the wrong disk, plug in the wrong cable, ignore system requirements etc - all with the best of intentions.

So what would happen if, say, a live application server is misconfigured to use a development database and then you unknowingly unleash real users upon it? You could spend a very long time troubleshooting it, or worse still it might actually work - and thinking about brownouts - how long will it be before you noticed? For users who'd attached to that node, where will all their changes be, and how will you merge that back into the 'real' live data?

Humans can also accidentally not do things which have consequences for our system too. Consider our feed example - perhaps we just forgot to renew the subscription, and so we're getting stale or no data even though the system has done everything it was designed to do. Hang on, who was in charge of updating those SSL certificates?

Perhaps we don't think about maintenance mistakes up front because whenever we build something, we always picture ourselves performing the operational tasks. And to us, the steps are obvious and we're performing them, in a simplified world in our heads, without any other distractions competing for our attention. Again - not real life.

And so...
All of these things can be monitored, tested for, and caught. In our forex example, you might check the age of the data every time you read the exchange rate value in preparation for a transaction, and fail it if it exceeds a certain threshold (or just watch the age in a separate process).

In our live server with test data example, you might mandate that systems and data sources state what mode they're in (test, live, demo, etc) in their connection string - better yet generate an alert if there is a state mismatch in the stack (or segment your network so communication is not possible).

The question isn't are there solutions; the question is how far is far enough?

As long as you think about failure modes in whatever way works for you, and make a pragmatic judgement on each risk using likelihood and impact to determine how many monitors and fail-safes it's worth building in, then you'll have done your job significantly better than the vast majority of engineers out there - and your customers will thank you for it with their business.

Root Cause Analysis

To help me kill some time at an airport (which seems to be my second job these days), let me reach into my wardrobe of soap-box issues and pick something out. Ah, root cause analysis, here we go.

In my opinion, proper root cause analysis is the most important part of any operational support process.

Having a professional, predictable response and the skills to restore service quickly are critical - but you have to ensure that your support processes don't stop there. If they do, then you're simply doomed to let history repeat itself, and this means more downtime, more reactive firefighting, and less satisfied customers.

Good root cause analysis takes into account the entire event - systemwide conditions, the teams response, the available data, the policies applied - not just the technical issue which triggered the fault, and looks for ways to reduce the likelihood of recurrence.

Doing root cause analysis properly can be expensive, because you don't need to get to the bottom of why it happened this time, it's why it keeps happening, and why the system was susceptible to the issue in the first place that you need to uncover to really add future value. Think of the time spent on it as an investment in availability, freeing up your team to work more strategically (as well as enjoy their jobs more), and happier users (which oddly seems to make happier engineers).

But what you learn by doing this isn't really worth the time you spend on it without the organizational discipline to follow up with real changes. If you're truly tracing issues back to their root, you'd be surprised how many are the result of a chain of events that could stretch right back to the earliest phases in projects. This needs commitment.

If you make money out of responding to problems then you'll probably want to ignore my advice. There is a whole industry of IT suppliers whose core business lives here, and while it's an admirable pursuit, don't take the habit with you when you join an internal team!

Fail More, Neo

If you expect everything to work all the time, if you believe everything will be perfect first time around, if you think everything you try and every idea you have will always be brilliant - then you are living in some kind of delusional hyper-fantasy. Stay there, trust me, it's better than out here.

But if you decide to take the red pill, and join us here in the real world, there are a few things you should know that will make your integration easier.

Firstly, machines have not taken over the world, and human beings are not just a kind of great big box of Duracell batteries to them. But this is mostly because we can't make machines awesome enough yet, see below...

Out here in the real world, we mostly learn by doing things (until we can make those machines that beam kung-fu directly into our brains). We try something out, observe the results, and then we do it again with some slight variations. Those little tweaks we make as we try again and again are based on the pain we feel each time it doesn't work out. We call this a feedback loop, and we've learned this way for thousands of years - if there was a better way to do it, a way we could just "get it right first time", then trust me, we'd be all over that by now!

We'll be honest with you - you can, even in the real world, get by with very little of such trial-and-error experience if you want. A well-established pattern of mundane mediocrity leading directly to obsolescence is readily available. In fact, you'd be surprised how popular a choice this actually is! Let's call it the grey pill.

Assuming you don't fancy the drab, lackluster, second-rate existence the grey pill guarantees, what else can you do with the humans under your command?

Firstly - and most importantly - allow them to try. Don't expect every idea to be killer, or everything to work out first time around. Allow - no wait - require experimentation and iteration. It is the number 1 way your humans will expand their understanding, and believe it or not, a string of failures is the shortest path to that one thing that does work brilliantly. Tom Watson, an old golf-playing human we have out here, once said “if you want to succeed, double your failure rate”.

You might notice that some of your humans seems a little reluctant to embrace the idea - especially those recently defrosted from 'grey pill' institutions. How can you spruce them up into lean, mean, mistake-making machines?

People often fear the consequences of failure more than failure itself. So the best course of action is to make the consequences of failure something to look forward to - not something to hide from, and cover up. Why not try celebrating failure? If one of your humans has an idea, tries it out, and then brings back some knowledge and experience to the rest of the team, then you are much better off than you were before. Make a big deal out of it, demonstrate to the rest of the team that it's OK to try. Stretching yourself won't be punished.

That doesn't mean rewarding scattergun bravado - what you're trying to encourage is a culture of balanced risk and methodical approach.

I like the old saying "fail fast, fail cheap" because as a statement, it gives permission to try new things, yet it is also prescribes some basic guidelines. Take the shortest path you can to discovering your idea doesn't work, and invest the minimum you need to in order to reach that same point. After all, you'll need those resources for your next idea.

So, thanks for joining us out here in the real world. We really hope you'll make the most of it by embracing failure and trying new things out, because this is the only path to discovery and success - oh and we'll never build those dominant supercomputers to have a nifty war with if we don't believe in innovating!

Maybe this post will be a failure. Maybe my message won't get into the dreamworld (where we don't believe in failure) or grey pill land (where we don't try just in case). But do you know what? I won't mind if it doesn't - because I will have just eliminated one of the ways it doesn't work, and that's a step forward...

Availability or Control?

In the web business, we usually consider availability to be paramount - and given that motivation, we're getting pretty good at things like graceful degradation and partial failure. But now that you've pulled your system apart and neatly isolated all the features, how do you cope with the situation where no service is preferable to partial service?

This can be true. Consider, if you will, a trading system operated by a team of risk managers. You have built the system to be fault tolerant and allow partial failures - and this usually works out great - but what happens if a failure in the infrastructure or application results in the risk managers no longer being able to administer the system? It's still running publicly (thanks to you awesome failure isolation) so customers are still buying and selling. You cant change your prices and respond to changing market conditions - uh oh - exposure. What do we do?

One answer is a word we don't like - especially if we just built a reasonably decoupled system - dependency. Yuck, but there is no shame in creating some intentional dependencies that support the business rules. If you never want to execute trades unless you can manage your position, then what is the advantage to running the trading system without the liability tool? Nothing - if anything it's an undesirable risk.

So draw up some service dependencies, or make the applications depend on their monitors at runtime. It might not appeal to how we'd like to run the system, but the truth is it accurately reflects how we'd like to run the business.

Scheduled Reboots and Natures Way

One of the basic aspects of a biological computing mindset is the appreciation that nothing lasts forever. Everything degrades, corrupts, and dies over time - and that is perfectly normal, because it's duly replaced by a fresh-faced youngster, eager to service the rest of the organism [system] from a nice, fresh cellular structure [empty memory space].

This applies to systems in exactly the same way as it does to organisms. How many issues can you recall where memory leaks, counter errors, and freaky edge conditions all occurred after servers have been running exactly X long, or when a service has processed more than Y connections. I'm sure we could swap tales of woe late into the evening.

This being the case, why do we feel this rottweiler-like dedication to keeping individual devices going for the longest possible duration? I think there is 2 sources; a kind of point scoring pride effect engendered by the output of the "uptime" command, and good old fashioned poor system design. Perhaps one even leads to the other...

So - we design systems poorly. If we want a product to be available, why do we build it in a way that it's availability depends upon a piece of tin that we accept is inherently unreliable? So now the application is the server. This means the only way we can increase its availability is by increasing the availability of the underlying hardware. Not only is this expensive, it's doomed to failure because, as we accepted, servers grow old. So we spend a lot of time and money trying to achieve something we already decided that we cannot. No wonder we're so excited when that uptime counter rolls over to a nice big number!

Do you know what would be better? Accepting that product availability - the uptime of the whole system overall - is what we're really reaching for, and besides, it's how our customers will measure us. Next we need to apply this philosophy to how we design systems, let go of our attachment to keeping individual servers on life support, and put together services that don't rely on any one node, network, or storage device in order to serve our customers.

If you can master that arcane art, then you'll be able to arbitrarily recycle resources, anytime, when there is absolutely nothing whatsoever wrong at all - because this helps keep it that way.

Oh and you'll never be that guy with the box thats been going so long he's scared to reboot it just in case it doesn't come back!

My Demotivational Poster

I just has to make one - and what better than my favorite topic (credit to whoever photoshopped tetris):

Helmuth von Moltke (a Prussian/German military strategist in the 1800's) once wisely observed that "no battle plan survives contact with the enemy".  I think you can say the same thing about software - no matter how elegant or simple your feature is, it will never survive contact with users.

That's why response and recovery plans are just as important as the code they are in place to support - you will need them sooner or later...

Tweet Tweet

Today I decided I'm going to play with twitter; so I signed up like so.  I can't really picture myself having the time to twit (or tweet?) every few hours, I signed up for slightly less conventional reasons...

It all stems from how I use this blog.  When I started blogging, I foresaw a channel for my original thoughts, a way for me to share my experience in the industry - real life problems and solutions from the world of running webscale engineering.  It was my opinion and what's worked for me; I didn't want to get into simply posting links to other people's opinion, unless I can substantially build on them, and thus add some value to the idea being discussed.  I wanted there to be some substance, some usefulness and some kind of conclusion to what you read here.

I have stayed true to this vision, but these days I am increasingly coming across content I want to share in a briefer 'check this link out' kind of format.  This of course poses the question; do I dilute the purpose of this blog by posting shorter, less meaningful messages with links or embedded external content or do I find another way (or, that oft-forgotten option we always have, do nothing)?

A tool that seemed fit for purpose to me was micro-blogging.  Small snippets of text pushed out as regularly as you see fit and a whole culture which prohibits verbosity (how will I cope).

So I'm going to play whack-a-link on twitter.  Anytime I see something I like or agree with (but don't want to more formally expand on) I'm going to demonstrate my support for it by posting the link into my feed.

For me this is one of those kind of experimental things - start using the technology and see what value emerges.

While we're talking about twitter I want to give the fail whale an honorable mention for achieving the pinnacle of error message accomplishment - being a popular sight.  No kidding.

Failure joins death and taxes in the hallowed halls of unavoidable inevitability.  I spend a lot of time working out how to detect it, avoid it, and recover from it but sooner or later it gets us all.  This is where architecture stops operations and customer service starts; personally I consider the likes of web server 404's and 500's to be the middle finger of the internet - if you can turn these into disarming, apologetic messages then you'll at least have a chance to keep your customers on your side while you work out your issues.

The fail whale is almost too good at this.  It's grown into some kind of phenomenon of it's own.  People have made fail whale models, you can buy fail whale t-shirts and mugs, and there is even a fan club.  Remember; this is a holding page they show when their site is down!

Teach Your Computer to Say No

As good engineers we always try to build systems that have enough capacity to service all our customers requests, all the time, anytime.  It's an admirable goal and very commercially sensible, but there is no such thing as unlimited capacity and sooner or later we'll get our scalability wrong, suffer partial failure or maybe it won't be cost effective to scale up to those rare usage peaks - what will the user experience be then?

By default it will be things like; page cannot be displayed, connection timed out or our staple favorites 404 and 500 - basically the middle finger of the internet.  You have to think about how your systems will behave under unexpected load and what you'd want your customers to see when it happens.  There are a few things you can keep in mind when building your products that, if properly addressed, will lead to a very different user experience when things get tough:

Customise your error messages

Those unexpected peaks or partial failures are always going to happen sooner or later, so why not serve up a nicely formatted (but lightweight) page with alternative links or customer service contact details rather than a nasty, cryptic error message?  It is so easy and cheap to do and makes such a huge difference to user perception that there really is no reason not to.

Queue or throttle excess traffic

What's better than a nicer error message?  Some kind of service - even if it doesn't match our ideals.  Queuing is a nice solution, it protects your in-flight users just like denial would and provides a better experience for those new arrivals who would otherwise push you over the edge and get an error.

A holding page, an alternative URL or maybe a nice countdown with an automatic refresh is all you need.  These things can be a little more time consuming to build because your application needs to have some awareness of it's environment if you expect it to make decisions about what to serve up based on remaining capacity.  There are some very easy ways to buy this back in hardware if you're using load balancers like F5, Netscaler, or Redline - a little experience will tell you how many users/connections/Mbps each node can tolerate, and you can configure an alternative page or redirect for anything above that threshold.  Depending on what you have installed this can even be served directly from the device cache, making it even lower impact for your over-busy system.  Queuing is just like it's real-life namesake; "line up here and wait for access to the system".

Throttling might suit certain applications better - particularly API's with a heavy non-human user population, as client applications might not understand queuing messages, holding pages or redirects returned from the oversubscribed feature but they'll be less sensitive to a general slowdown.  Throttling, as I use it here, is about reducing the maximum share of total system capacity any individual can consume in the hope that this will allow more total individuals to use the system concurrently.  The theory is that once we hit a certain threshold, if we flatten out everyone's usage, the big consumers will be held back a little (but still served) and the rest of us can still have a turn too.  Like most ideas here, this can be implemented in a variety of places in your stack; queries/sec or concurrent data sets searchable in the database, concurrent logons in the middle tier, TPS in the application, GETs/POSTs on the front end or even number of concurrent connections per unique client IP address at the network edge.

Where is the right place to impose planned limits?  I'd suggest multipoint coverage, but to start with think about your most critical constraint (what dies first as usage climbs?) and then do it at the layer above that one!

Fail gracefully

This is one of the toughest things to do with a distributed system; your features need to be aware of their environment both locally and on remote instances and they need a way to bring up/take down/recycle themselves and each other.  If your system suffers a partial failure, or load is climbing towards the point marked "instability", then you need to make a decision.  Do you bring up or repurpose more nodes to handle the same feature?  Redirect a percentage of requests to another cluster/DC or trigger any of the other techniques we talked about above?  Great if you can, but if you can't, it might be time to use those [refactored to be friendly] error messages.

Turning away users by conscious decision may rub us up the wrong way at first glance, but depending on how your system behaves under excessive load it might actually be for the best.  If you are at maximum capacity and serving all currently connected users, would new user connections result in degradation of service to those already in the system?  Maybe saying no to additional connections might be frustrating for those users being turned away, but you have to weigh that against kicking off users midway through transactions, or worse still, crashing the entire system!

Graceful failure is essentially the art of predicting the immediate future of your system and handling what are likely to be excess users/transactions/connections in a premeditated fashion.  First you need to know what your thresholds are; then you need very fine resolution instrumentation to tell you how close you are to them on a second-by-second basis, and finally, you need an automated way to respond to impending trouble - because humans are too slow to prevent poor customer service in busy systems; we're better at cleaning up once the trouble has been and gone.

For example let's say you're aware of a memory utilization threshold or a maximum number of users/web node - wouldn't it be better to actively restart a service or show a holding page before you lost control of the stack?  If you haven't managed to build in some kind of queuing or throttling then you might be showing an error or denying service but is that better than losing the whole system.

The last part of graceful failure somewhat overlaps with recovery-oriented computing; make sure that, upon death, the last thing you ever do is take a snapshot of what you were doing and what the environment was like.  If your processes do this, then you are able to have watchdog processes (or monitoring systems) that know whether or not it's safe to restart that failed instance (or take an alternative action based on the data), you'll have an easier time diagnosing faults, and the data generated will help you keep a rolling benchmark of the thresholds in a system with a high rate of change.

S**t Happens

I talk a lot about failure, how to build for it and recover from it.  Of all the things that will happen to your system during its lifetime failure of some sort is one of the few inevitable events.

A lot can go wrong with computers, but surely their best-known weakness has to be their fundamental incompatibility with water.

Focusing on building systems that survive individual node failure is an excellent discipline, but as you can see from that clip, you can't count on your datacenter to always be there.  That means distributing your system across servers in the same location will protect you from a number of (the most common) failure scenarios but if it's really, really important that you are always up then it needs to be in more than one place.

Think electricity.  Think connectivity.  Think geography.

Rack Mount 1, Technician 0

Failure is coming to get you, but we're getting better at predicting the scenarios and coding for them.  We think a lot about servers dying, losing network connectivity, power cuts, and how to respond to critical bugs.  These things are essentially unexpected technical events, but there is a whole other category at play in real life - human error.

Imagine this server has your data on it...

What will your customers see while that gets put back together?  How are you going to get the data back?

Despite the comedy value of that clip, this is exactly the sort of thing that happens in real life - people make mistakes.  But even when this kind of maintenance is less clownishly executed, it still needs to happen - and you need to decide what effect you're going to let planned maintenance events have on your revenue stream.

There is a lesson in here if you look closely...

We had a tough weekend but if we take the right things away from it then it will be one of the cheapest lessons we could pay for...

A while back we launched a new product; starting price.  The concept is pretty simple and analogous to it's parent product (the sports exchange) where for every event backers are matched against layers.  In the case of starting price bets, we are matching starting price backers (who have specified a stake) and exchange backers with starting price layers (who have specified a liability) and exchange layers.  So just like traditional SP but with our unique peer-to-peer slant on it.  There is one more big benefit I like, SP lets you set a persistent bet at the beginning of an market which stays open once the market goes in-play (these used to be voided) which is big for us because that can be tens of thousands of potential matches depending on the event.

So what went wrong?  Depends who you ask.  Let's say I ask my monitoring systems - nothing.  How about my sysadmins - nothing.  Intrusion detection - nothing.  If I ask my developers, testers, mathematician - still nothing.  Anecdotally (the currency of monitoring) I can say nothing went wrong at all yet I am still not happy.  By "I" I really mean the business and the business wasn't happy because our SP was being calculated at around 50% of the industry benchmark - on the one hand if that's the true market value then that's the true market value but on the other hand it just isn't how we'd planned that the product would perform.

So here is what I think went wrong - we built a sunny day system.  Product managers will always write specs based on what they want to happen, the journey they have planned for their users.  They'll hardly ever write about what they don't want to happen or think about an alternative journey a user might take through the system.  It is just as important to think about what else people might do, through legitimate or malicious use, with your functionality - if the only input to your design is the desired behavior then you're doomed (unless you are unfeasibly lucky).  On top of all this you should also add common failure scenarios and some sensible behavior under exceptions and variable performance (latency, disk space etc).

What was our newbie mistake?  Most people that are new to a trading system back only and it takes them a long time to build up the courage to try some lay betting.  Good old fashioned assumption; "people will use SP to back and lay" rather than "most people will use SP to back only" and I can't help thinking that had we considered this simple piece of human behavior (that we considered undesirable) we could have built in some rule - min and max thresholds or distance from trend etc.

That's software in the real world.

The Great Divide

I'm into availability at the moment, so much so that I'm leading an organization-wide change initiative that targets cultural, technical and process habits (busting established, forming new) in order to deliver a more consistent experience.  Our goal is to always have something to offer our customers regardless of any maintenance we're performing or failures we're experiencing.

This is how I discovered The Great Divide.  The namesake of this post is that gap between the infrastructure and the software - because of which we offer our product to our users much less often than we could.

Here is how it works:

We've got a pairs of firewalls that can fail over while maintaining session state.  We've got tiers of load balancers that can reroute traffic around down network devices.  We've got clustered databases that can move active systems between nodes in a couple of minutes.

But guess what else we've got?

We've got applications that lose session information without contiguous sequence numbers.  We've got applications that cant match users to activity if their traffic suddenly comes from another IP address.  We've got applications that depend so heavily on their databases that death occurs within a few seconds of separation.

Damn.

You don't get any partial credit in product uptime - your customers will not award you a bonus point if your site is down but your servers are up.  If they cant log in they cant log in, if they cant place orders they cant place orders; they're quite a binary bunch.

For us product = infrastructure + software + operational know-how to run it.  We need to stop worrying about server/network availability and start worrying about product availability - because guess what, that's what our customers are measuring us on.

Close that gap and let your customers see the benefit of those cool devices.

A Good Quote

I talk a lot about failure in systems - it's inevitability and the things we can do to minimize it's impact and recover in a quick, automated fashion (note I never said "prevent" failure).

I recently came across a good quote on the topic that I think is really punchy:

"failure is not an option - it's bundled with your software"

I would credit it's author but I can't seem to work out where it came from - I've seen it tacked on the end of a few people's forum posts. I can't help but think they meant it a bit more cynically than I'm taking it but it's a good thinking point when you're delivering online services.